Unable to readPixels into numpy array also can’t export product even with 1100*1200 area of a sentinel 1 tile.
I changes all the parameters, showing bellow:
VM parameter:
-Xmx5900m -Xms2048m -XX:+AggressiveOpts -Xverify:none -Dnetbeans.mainclass=org.esa.snap.main.Main -Dsun.java2d.noddraw=true -Dsun.awt.nopixfmt=true -Dsun.java2d.dpiaware=false
How much RAM does your system have? You can watch system memory usage with Windows Task Manager to see if your system is running out of memory space. SNAP’s processing generally processes small sections of the arrays at a time, so can work with smaller -Xmx values. Numpy arrays are stored in RAM. It is possible that your memory settings don’t leave enough space for Python’s data, so when numpy can’t get space for the array, Java reports “OutOfMemory”.
If possible, you should use SNAP Java functions to process arrays in snappy. With numpy, there are techniques to reduce memory requirements, for example, using <op>= to rewrite:
@gnwiii I have 8 GB of RAM in my machine. I khow with this size of RAM I can’t work with full tile. So, I run a spatial subset to reduce the data volume and now I have a tile of 1100*1200 pixels and 4 bands [“Sigma0_VH”,“Sigma0_VV”,“Local Incident angel”,"Incident angle from Ellipsoid "]. But still now It failed to readPixels of a single band. Also unable to write product in disk. I already tried with smaller -Xmx values. Please give me any solution for this problem. I can’t increase my Physical RAM. Is it use the Physical RAM when I am writing a Product? Please say me what would be the smaller amount of RAM in -Xmx without any other errors?
Can you elaborate this to me ? and any other advance guidance for snap functions?
One more thing how can I write a single band from a image using ProductIO.writeProduct() function?
Please help.
I have 8G in my laptop, and I’m using SNAP with the default memory settings. This may not leave enough memory for numpy, especially if you have other programs running. With SNAP, the Java algorithms can work the original spatial extent because it only processes a small port of the image at a time. Numpy, however, does work with the full input array size. My system has very little memory left while a
browser and the snappy program is running, so you probably need to run numpy separately after saving the subset.
Are you able to run the snappy examples? snappy_subset.py writes a subset to a BEAM DIMAP file. You should be able to modify it to save NetCDF4-CF format,
which is very well supported in Python.
@gnwiii Yes I tried to run the example file. but the datasets used for test are not available in my snappy directory. No not tried it with NetCDF4-CF format because I am not so much familiar with this format. Ok I will try with this format. Can you please share the default snap configuration in memory portion for your pc? thank you.
You need to download “snappy test data”, unzip, rename MER_RR__1PNRAL20100426_154828_000003662088_00498_42636_0001.N1 to MER_RR__1P.N1, and copy to the snappy\tests folder (this needs to added to the snappy configuration instructions). Make sure you can run the tests before moving on to the examples. If you have a problem with any of these files then lots of other people have the same files and may be able to help.
@gnwiii It is really surprised me every time. I can export 4 bands each containing 4256x2584 pixels but can’t export one band of Sentinel 1 tile containing 3568x2564 pixels?
Seriously! are you sure the problem is due to Insufficient RAM?
How it is possible?
Different sensors and processing levels often store data very differently (differing numerical representations), bands in separate arrays or interleaved in one array (each “pixel” is a vector), so time and memory demands for similar operations can vary widely.
I am saying about same image one is cropped with 10m resolution and another is full tile of same sentinel-1A image in 100m resolution and single band Sigma0_VH.
I thought, the problem is generating due to 8 GB RAM in my PC. But now I am working on a PC with 16 GB RAM but still now unable to export 16-17 MB terrain corrected tiles, snaphu export taking time over 45 min while my tile size 1117*658 pixels. I am working with 11 GB of max RAM for snappy. Can’t understand why??? Tell me how we can process data with snappy or snap? While QGIS can handle a full tile image of sentinel 2 surface ref. in 8 GB of RAM, why SNAP/SNAPPY can’t handle a 16-17 MB of data? Even with 16 GB of RAM!!!
How we can handle and analysis S1 data??? Have you any solution??
You need to describe the workflow in enough detail for someone else to reproduce the problem. Are you able to run the snappy tests and examples? Many people can run those without encountering problems.
@gnwiii
I run the test_snappy_mem file
here is the result
> test_readProduct_and_readPixels_mem
i = 0
i = 1
i = 2
i = 3
i = 4
i = 5
i = 6
i = 7
i = 8
i = 9
i = 10
i = 11
i = 12
i = 13
i = 14
i = 15
i = 16
i = 17
i = 18
i = 19
i = 20
i = 21
i = 22
i = 23
i = 24
i = 25
i = 26
i = 27
i = 28
i = 29
i = 30
i = 31
i = 32
i = 33
i = 34
i = 35
i = 36
i = 37
i = 38
i = 39
i = 40
i = 41
i = 42
i = 43
i = 44
i = 45
i = 46
i = 47
i = 48
i = 49
i = 50
i = 51
i = 52
i = 53
i = 54
i = 55
i = 56
i = 57
i = 58
i = 59
i = 60
i = 61
i = 62
i = 63
i = 64
i = 65
i = 66
i = 67
i = 68
i = 69
i = 70
i = 71
i = 72
i = 73
i = 74
i = 75
i = 76
i = 77
i = 78
i = 79
i = 80
i = 81
i = 82
i = 83
i = 84
i = 85
i = 86
i = 87
i = 88
i = 89
i = 90
i = 91
i = 92
i = 93
i = 94
i = 95
i = 96
i = 97
i = 98
i = 99
i = 100
i = 101
i = 102
i = 103
i = 104
i = 105
i = 106
i = 107
i = 108
i = 109
i = 110
i = 111
i = 112
i = 113
i = 114
i = 115
i = 116
i = 117
i = 118
i = 119
i = 120
i = 121
i = 122
i = 123
i = 124
i = 125
i = 126
i = 127
i = 128
i = 129
i = 130
i = 131
i = 132
i = 133
i = 134
i = 135
i = 136
i = 137
i = 138
i = 139
i = 140
i = 141
i = 142
i = 143
i = 144
i = 145
i = 146
i = 147
i = 148
i = 149
i = 150
i = 151
i = 152
i = 153
i = 154
i = 155
i = 156
i = 157
i = 158
i = 159
i = 160
i = 161
i = 162
i = 163
i = 164
i = 165
i = 166
i = 167
i = 168
i = 169
i = 170
i = 171
i = 172
i = 173
i = 174
i = 175
i = 176
i = 177
i = 178
i = 179
i = 180
i = 181
i = 182
i = 183
i = 184
i = 185
i = 186
i = 187
i = 188
i = 189
i = 190
i = 191
i = 192
i = 193
i = 194
i = 195
i = 196
i = 197
i = 198
i = 199
i = 200
i = 201
i = 202
i = 203
i = 204
i = 205
i = 206
i = 207
i = 208
i = 209
i = 210
i = 211
i = 212
i = 213
i = 214
i = 215
i = 216
i = 217
i = 218
i = 219
i = 220
i = 221
i = 222
i = 223
i = 224
i = 225
i = 226
i = 227
i = 228
i = 229
i = 230
i = 231
i = 232
i = 233
i = 234
i = 235
i = 236
i = 237
i = 238
i = 239
i = 240
i = 241
i = 242
i = 243
i = 244
i = 245
i = 246
i = 247
i = 248
i = 249
i = 250
i = 251
i = 252
i = 253
i = 254
i = 255
i = 256
i = 257
i = 258
i = 259
i = 260
i = 261
i = 262
i = 263
i = 264
i = 265
i = 266
i = 267
i = 268
i = 269
i = 270
i = 271
i = 272
i = 273
i = 274
i = 275
i = 276
i = 277
i = 278
i = 279
i = 280
i = 281
i = 282
i = 283
i = 284
i = 285
i = 286
i = 287
i = 288
i = 289
i = 290
i = 291
i = 292
i = 293
i = 294
i = 295
i = 296
i = 297
i = 298
i = 299
i = 300
i = 301
i = 302
i = 303
i = 304
i = 305
i = 306
i = 307
i = 308
i = 309
i = 310
i = 311
i = 312
i = 313
i = 314
i = 315
i = 316
i = 317
i = 318
i = 319
i = 320
i = 321
i = 322
i = 323
i = 324
i = 325
i = 326
i = 327
i = 328
i = 329
i = 330
i = 331
i = 332
i = 333
i = 334
i = 335
i = 336
i = 337
i = 338
i = 339
i = 340
i = 341
i = 342
i = 343
i = 344
i = 345
i = 346
i = 347
i = 348
i = 349
i = 350
i = 351
i = 352
i = 353
i = 354
i = 355
i = 356
i = 357
i = 358
i = 359
i = 360
i = 361
i = 362
.i = 363
i = 364
----------------------------------------------------------------------
Ran 1 test in 24.887s
OK
runfile(‘C:/Users/subha/Desktop/snappy/tests/test_snappy_perf.py’, wdir=‘C:/Users/subha/Desktop/snappy/tests’)
…Band.readPixels(): w = 221 , dtype=np.float32: 184 calls in 3.9975643157958984 ms, that is 0.02184461374751857 ms per call
Band.readValidMask(): w = 221 , dtype=np.bool: 184 calls in 29.00242805480957 ms, that is 0.15848321341425994 ms per call
runfile(‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples/snappy_bmaths.py’, wdir=‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples’)
usage: C:\Users\subha\anaconda3\Lib\site-packages\snappy\examples\snappy_bmaths.py
An exception has occurred, use %tb to see the full traceback.
> runfile(‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples/snappy_bmaths.py’, wdir=‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples’)
> usage: C:\Users\subha\anaconda3\Lib\site-packages\snappy\examples\snappy_bmaths.py
> An exception has occurred, use %tb to see the full traceback.
>
>
>
> SystemExit: 1
is missing the args="<filename_with_path>" entry. If you have been trying to run snappy scripts using runfile in Spyder you should try running the script in a terminal. It is not unusual to have memory issues when using runfile.