Java OutOfMemory error snappy

Unable to readPixels into numpy array also can’t export product even with 1100*1200 area of a sentinel 1 tile.
I changes all the parameters, showing bellow:
VM parameter:
-Xmx5900m -Xms2048m -XX:+AggressiveOpts -Xverify:none -Dnetbeans.mainclass=org.esa.snap.main.Main -Dsun.java2d.noddraw=true -Dsun.awt.nopixfmt=true -Dsun.java2d.dpiaware=false

Cache size: 8540

in snappy.ini:
[DEFAULT]
snap_home: C:/Program Files/snap
java_class_path: target/classes
java_library_path: target/lib/amd64
java_options: -Djava.awt.headless=true
java_max_mem: 6G
debug: False

jpyconfig.py:
jvm_maxmem = 6G

Can anyone guide me where is the problem? I am unable to read pixel or write product.

How much RAM does your system have? You can watch system memory usage with Windows Task Manager to see if your system is running out of memory space. SNAP’s processing generally processes small sections of the arrays at a time, so can work with smaller -Xmx values. Numpy arrays are stored in RAM. It is possible that your memory settings don’t leave enough space for Python’s data, so when numpy can’t get space for the array, Java reports “OutOfMemory”.

If possible, you should use SNAP Java functions to process arrays in snappy. With numpy, there are techniques to reduce memory requirements, for example, using <op>= to rewrite:

x = x + const*y

to

y *= const
x += y

@gnwiii I have 8 GB of RAM in my machine. I khow with this size of RAM I can’t work with full tile. So, I run a spatial subset to reduce the data volume and now I have a tile of 1100*1200 pixels and 4 bands [“Sigma0_VH”,“Sigma0_VV”,“Local Incident angel”,"Incident angle from Ellipsoid "]. But still now It failed to readPixels of a single band. Also unable to write product in disk. I already tried with smaller -Xmx values. Please give me any solution for this problem. I can’t increase my Physical RAM. Is it use the Physical RAM when I am writing a Product? Please say me what would be the smaller amount of RAM in -Xmx without any other errors?

Can you elaborate this to me ? and any other advance guidance for snap functions?
One more thing how can I write a single band from a image using ProductIO.writeProduct() function?
Please help.

I have 8G in my laptop, and I’m using SNAP with the default memory settings. This may not leave enough memory for numpy, especially if you have other programs running. With SNAP, the Java algorithms can work the original spatial extent because it only processes a small port of the image at a time. Numpy, however, does work with the full input array size. My system has very little memory left while a
browser and the snappy program is running, so you probably need to run numpy separately after saving the subset.

Are you able to run the snappy examples? snappy_subset.py writes a subset to a BEAM DIMAP file. You should be able to modify it to save NetCDF4-CF format,
which is very well supported in Python.

@gnwiii Yes I tried to run the example file. but the datasets used for test are not available in my snappy directory. No not tried it with NetCDF4-CF format because I am not so much familiar with this format. Ok I will try with this format. Can you please share the default snap configuration in memory portion for your pc? thank you.

You need to download “snappy test data”, unzip, rename MER_RR__1PNRAL20100426_154828_000003662088_00498_42636_0001.N1 to MER_RR__1P.N1, and copy to the snappy\tests folder (this needs to added to the snappy configuration instructions). Make sure you can run the tests before moving on to the examples. If you have a problem with any of these files then lots of other people have the same files and may be able to help.

jpyconfig.properties (453 Bytes) snappy.ini (216 Bytes)

1 Like

@gnwiii It is really surprised me every time. I can export 4 bands each containing 4256x2584 pixels but can’t export one band of Sentinel 1 tile containing 3568x2564 pixels?

Seriously! are you sure the problem is due to Insufficient RAM?
How it is possible?

Different sensors and processing levels often store data very differently (differing numerical representations), bands in separate arrays or interleaved in one array (each “pixel” is a vector), so time and memory demands for similar operations can vary widely.

1 Like

I am saying about same image one is cropped with 10m resolution and another is full tile of same sentinel-1A image in 100m resolution and single band Sigma0_VH.

I thought, the problem is generating due to 8 GB RAM in my PC. But now I am working on a PC with 16 GB RAM but still now unable to export 16-17 MB terrain corrected tiles, snaphu export taking time over 45 min while my tile size 1117*658 pixels. I am working with 11 GB of max RAM for snappy. Can’t understand why??? Tell me how we can process data with snappy or snap? While QGIS can handle a full tile image of sentinel 2 surface ref. in 8 GB of RAM, why SNAP/SNAPPY can’t handle a 16-17 MB of data? Even with 16 GB of RAM!!!
How we can handle and analysis S1 data??? Have you any solution??

You need to describe the workflow in enough detail for someone else to reproduce the problem. Are you able to run the snappy tests and examples? Many people can run those without encountering problems.

Ok, I am generating displacement map using snappy. that’s it. the problem is java heap space error. I already told about my image size.

Do the snappy tests and examples work on your system?

You need to tell us more so someone can reproduce the problem without guessing about details:

  1. OS and Python details (since you changed systems) so someone who has a similar system can try your calculation

  2. input data If you have done some processing you need to descriebe what was done or put the data on a public server (e.g., google drive).

  3. Python script

test_product_file = ‘./MER_RR__1P.N1’
this file is not availabe in my snappy folder
import unittest
import array
import sys

import numpy as np

import snappy

JAI = snappy.jpy.get_type('javax.media.jai.JAI')
JAI.getDefaultInstance().getTileCache().setMemoryCapacity(128 * 1000 * 1000)

test_product_file = './MER_RR__1P.N1'

class TestBeamIO(unittest.TestCase):

    def setUp(self):
        self.product = snappy.ProductIO.readProduct(test_product_file)
        self.assertIsNotNone(self.product)


    def tearDown(self):
        self.product.dispose()


    def test_getProductReader(self):
        #print('Band.mro =', snappy.Band.mro())
        reader = self.product.getProductReader()
        self.assertIsNotNone(reader)
        # TODO: fix me: AttributeError: 'org.esa.snap.core.dataio.ProductReader' object has no attribute 'getClass'
        #print('ProductReader.mro =', type(reader).mro())
        #readerClass = reader.getClass()
        #self.assertEqual(readerClass.getName(), '??')


    def test_getBandNames(self):
        names = self.product.getBandNames()
        self.assertEqual(len(names), 17)
        self.assertEqual(names[0], 'radiance_1')
        self.assertEqual(names[14], 'radiance_15')
        self.assertEqual(names[15], 'l1_flags')
        self.assertEqual(names[16], 'detector_index')


    def test_getSceneRasterWidthAndHeight(self):
        w = self.product.getSceneRasterWidth()
        h = self.product.getSceneRasterHeight()
        self.assertEqual(w, 1121)
        self.assertTrue(h > 0)


    def test_readPixels_with_java_array(self):
        w = self.product.getSceneRasterWidth()
        h = self.product.getSceneRasterHeight()
        b = self.product.getBand('radiance_13')
        a = snappy.jpy.array('float', w)
        b.readPixels(0, 0, w, 1, a)
        self.assertTrue(a[0] == 0.0)
        self.assertTrue(0 < a[100] < 200)


    def test_readPixels_with_python_array(self):
        if sys.version_info < (3, 0,):
            # Test only on Python 3.x, as the 2.x array type does not support the new buffer interface
            return
            
        w = self.product.getSceneRasterWidth()
        h = self.product.getSceneRasterHeight()
        b = self.product.getBand('radiance_13')
        a = array.array('f', w * [0])
        b.readPixels(0, 0, w, 1, a)
        self.assertTrue(a[0] == 0.0)
        self.assertTrue(0 < a[100] < 200)


    def test_readPixels_with_numpy_array(self):
        w = self.product.getSceneRasterWidth()
        h = self.product.getSceneRasterHeight()
        b = self.product.getBand('radiance_13')
        a = np.zeros(w, dtype=np.float32)
        b.readPixels(0, 0, w, 1, a)
        self.assertTrue(a[0] == 0.0)
        self.assertTrue(0 < a[100] < 200)


    def test_readValidMask_with_numpy_array(self):
        w = self.product.getSceneRasterWidth()
        h = self.product.getSceneRasterHeight()
        b = self.product.getBand('radiance_13')
        a = np.zeros(w, dtype=np.int8)
        #snappy.jpy.diag.flags = snappy.jpy.diag.F_ALL
        b.readValidMask(0, 0, w, 1, a)
        #snappy.jpy.diag.flags = snappy.jpy.diag.F_OFF
        self.assertEqual(a[0], 0)
        self.assertEqual(a[100], 1)


if __name__ == '__main__':
    unittest.main()
..FFFFF
======================================================================
FAIL: test_getSceneRasterWidthAndHeight (__main__.TestBeamIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\subha\Desktop\snappy\tests\test_snappy_product.py", line 47, in test_getSceneRasterWidthAndHeight
    self.assertEqual(w, 1121)
AssertionError: 221 != 1121

======================================================================
FAIL: test_readPixels_with_java_array (__main__.TestBeamIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\subha\Desktop\snappy\tests\test_snappy_product.py", line 57, in test_readPixels_with_java_array
    self.assertTrue(a[0] == 0.0)
AssertionError: False is not true

======================================================================
FAIL: test_readPixels_with_numpy_array (__main__.TestBeamIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\subha\Desktop\snappy\tests\test_snappy_product.py", line 81, in test_readPixels_with_numpy_array
    self.assertTrue(a[0] == 0.0)
AssertionError: False is not true

======================================================================
FAIL: test_readPixels_with_python_array (__main__.TestBeamIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\subha\Desktop\snappy\tests\test_snappy_product.py", line 71, in test_readPixels_with_python_array
    self.assertTrue(a[0] == 0.0)
AssertionError: False is not true

======================================================================
FAIL: test_readValidMask_with_numpy_array (__main__.TestBeamIO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\subha\Desktop\snappy\tests\test_snappy_product.py", line 93, in test_readValidMask_with_numpy_array
    self.assertEqual(a[0], 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 7 tests in 6.284s

FAILED (failures=5)

I tried with MER_FRS_L1B_SUBSET.dim file in snappy testdata folder this is the result

@gnwiii
I run the test_snappy_mem file
here is the result
> test_readProduct_and_readPixels_mem

i =  0
i =  1
i =  2
i =  3
i =  4
i =  5
i =  6
i =  7
i =  8
i =  9
i =  10
i =  11
i =  12
i =  13
i =  14
i =  15
i =  16
i =  17
i =  18
i =  19
i =  20
i =  21
i =  22
i =  23
i =  24
i =  25
i =  26
i =  27
i =  28
i =  29
i =  30
i =  31
i =  32
i =  33
i =  34
i =  35
i =  36
i =  37
i =  38
i =  39
i =  40
i =  41
i =  42
i =  43
i =  44
i =  45
i =  46
i =  47
i =  48
i =  49
i =  50
i =  51
i =  52
i =  53
i =  54
i =  55
i =  56
i =  57
i =  58
i =  59
i =  60
i =  61
i =  62
i =  63
i =  64
i =  65
i =  66
i =  67
i =  68
i =  69
i =  70
i =  71
i =  72
i =  73
i =  74
i =  75
i =  76
i =  77
i =  78
i =  79
i =  80
i =  81
i =  82
i =  83
i =  84
i =  85
i =  86
i =  87
i =  88
i =  89
i =  90
i =  91
i =  92
i =  93
i =  94
i =  95
i =  96
i =  97
i =  98
i =  99
i =  100
i =  101
i =  102
i =  103
i =  104
i =  105
i =  106
i =  107
i =  108
i =  109
i =  110
i =  111
i =  112
i =  113
i =  114
i =  115
i =  116
i =  117
i =  118
i =  119
i =  120
i =  121
i =  122
i =  123
i =  124
i =  125
i =  126
i =  127
i =  128
i =  129
i =  130
i =  131
i =  132
i =  133
i =  134
i =  135
i =  136
i =  137
i =  138
i =  139
i =  140
i =  141
i =  142
i =  143
i =  144
i =  145
i =  146
i =  147
i =  148
i =  149
i =  150
i =  151
i =  152
i =  153
i =  154
i =  155
i =  156
i =  157
i =  158
i =  159
i =  160
i =  161
i =  162
i =  163
i =  164
i =  165
i =  166
i =  167
i =  168
i =  169
i =  170
i =  171
i =  172
i =  173
i =  174
i =  175
i =  176
i =  177
i =  178
i =  179
i =  180
i =  181
i =  182
i =  183
i =  184
i =  185
i =  186
i =  187
i =  188
i =  189
i =  190
i =  191
i =  192
i =  193
i =  194
i =  195
i =  196
i =  197
i =  198
i =  199
i =  200
i =  201
i =  202
i =  203
i =  204
i =  205
i =  206
i =  207
i =  208
i =  209
i =  210
i =  211
i =  212
i =  213
i =  214
i =  215
i =  216
i =  217
i =  218
i =  219
i =  220
i =  221
i =  222
i =  223
i =  224
i =  225
i =  226
i =  227
i =  228
i =  229
i =  230
i =  231
i =  232
i =  233
i =  234
i =  235
i =  236
i =  237
i =  238
i =  239
i =  240
i =  241
i =  242
i =  243
i =  244
i =  245
i =  246
i =  247
i =  248
i =  249
i =  250
i =  251
i =  252
i =  253
i =  254
i =  255
i =  256
i =  257
i =  258
i =  259
i =  260
i =  261
i =  262
i =  263
i =  264
i =  265
i =  266
i =  267
i =  268
i =  269
i =  270
i =  271
i =  272
i =  273
i =  274
i =  275
i =  276
i =  277
i =  278
i =  279
i =  280
i =  281
i =  282
i =  283
i =  284
i =  285
i =  286
i =  287
i =  288
i =  289
i =  290
i =  291
i =  292
i =  293
i =  294
i =  295
i =  296
i =  297
i =  298
i =  299
i =  300
i =  301
i =  302
i =  303
i =  304
i =  305
i =  306
i =  307
i =  308
i =  309
i =  310
i =  311
i =  312
i =  313
i =  314
i =  315
i =  316
i =  317
i =  318
i =  319
i =  320
i =  321
i =  322
i =  323
i =  324
i =  325
i =  326
i =  327
i =  328
i =  329
i =  330
i =  331
i =  332
i =  333
i =  334
i =  335
i =  336
i =  337
i =  338
i =  339
i =  340
i =  341
i =  342
i =  343
i =  344
i =  345
i =  346
i =  347
i =  348
i =  349
i =  350
i =  351
i =  352
i =  353
i =  354
i =  355
i =  356
i =  357
i =  358
i =  359
i =  360
i =  361
i =  362
.i =  363
i =  364

----------------------------------------------------------------------
Ran 1 test in 24.887s

OK

@gnwiii

runfile(‘C:/Users/subha/Desktop/snappy/tests/test_snappy_perf.py’, wdir=‘C:/Users/subha/Desktop/snappy/tests’)
…Band.readPixels(): w = 221 , dtype=np.float32: 184 calls in 3.9975643157958984 ms, that is 0.02184461374751857 ms per call
Band.readValidMask(): w = 221 , dtype=np.bool: 184 calls in 29.00242805480957 ms, that is 0.15848321341425994 ms per call


Ran 2 tests in 0.198s

OK

runfile(‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples/snappy_bmaths.py’, wdir=‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples’)
usage: C:\Users\subha\anaconda3\Lib\site-packages\snappy\examples\snappy_bmaths.py
An exception has occurred, use %tb to see the full traceback.

SystemExit: 1

Look back in this thread (May 3) for the link to the snappy test data and installation instructions.

Your example:

> runfile(‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples/snappy_bmaths.py’, wdir=‘C:/Users/subha/anaconda3/Lib/site-packages/snappy/examples’)
> usage: C:\Users\subha\anaconda3\Lib\site-packages\snappy\examples\snappy_bmaths.py
> An exception has occurred, use %tb to see the full traceback.
>
>
>
> SystemExit: 1

is missing the args="<filename_with_path>" entry. If you have been trying to run snappy scripts using runfile in Spyder you should try running the script in a terminal. It is not unusual to have memory issues when using runfile.