Snappy ProductIO.readProduct() Java OutOfMemoryError

snap
#1

Dear all,

while using the ProductIO module of snappy in my Python environment (PyCharm), the following RuntimeError arises while executing ProductIO.readProduct():
Traceback (most recent call last):
File “”, line 1, in
RuntimeError: java.lang.OutOfMemoryError

What I’ve tried so far to find a solution:

  1. Increase Java max heap size in Java Runtime Environment
  2. Increase PyCharm VM max heap size
  3. Edit java heap space in snap.conf, snappy.ini and jpyconfig.py

Nothing of these changes solved the problem. What I also checked were processes running on my computer. I found several opj_dump.exe which occupied mostly all of my CPU. The opj_dump files are located in the following path: .snap\auxdata\openjpeg\6.0.4\openjpeg-2.1.0-win64\bin

Some more information on my PC: I use Python 3.7, my SNAP installation is version 6 (updated today) and my system software is Windows 10 (x64, 32 GB RAM).

Does somebody have any explanation or idea to solving this issue?

Any help is highly appreciated! :blush:

Many thanks,
Cheers,
Kim

#2

besides RAM, one of the limiting factors can be the reading/writing speed of your hard drive.
If your computer has a SSD drive, it is worth trying to put it there and run again, for example under C:\S1

#3

Dear Andreas,

thank you for your quick reply.

I already tried by copying one S2-SAFE-folder to C: but the MemoryError occured as well. Sorry for not having mentioned before.

Cheers!

#4

no worries, I just wanted to go sure.
Are you using a Sentinel-2 tile or one of the old “fullsize” products?

#5

I’m using a S2B scene from 24/04/2019, which I atmospherically corrected with sen2cor. What I try to read in is a 20 m band as JPEG2000 (size is 10,1 MB).

#6

yes, that should be possible :sweat_smile:
Let’s wait for any of the snappy experts here.

1 Like
#7

Hi

To find out why the memory issue persists, could you add the following at the beginning of your script?

Runtime = jpy.get_type(‘java.lang.Runtime’)
heap_size = Runtime.getRuntime().maxMemory() / (1024*1024)
print(heap_size, "MB)

Does the value match the one you have specified in snappy.ini?

The opj_dump.exe is used to decompress the jpeg2000 files. Hence, it is okay if they use the CPU a lot.

#8

Hey Marco,

thanks for your idea. 14564.0 MB are returned and I specified 16 GB in the snappy.ini file. Seems that it fits.
Regarding the opj_dump.exe: is it also relatable that more than one of this process are running?

Just adding my code, maybe it helps anything in finding a solution on the memory error:

from snappy import ProductIO
import jpy
import os

Runtime = jpy.get_type('java.lang.Runtime')
heap_size = Runtime.getRuntime().maxMemory() / (1024*1024)
print(heap_size, "MB")

os.chdir(r"\IMG_DATA\R20m")
inputImage = r"T16PGR_20190429T155911_B05_20m.jp2"
product = ProductIO.readProduct(inputImage)

Thank you!

Added 16th October:

When I import snappy by executing

import snappy

I get the following error message:

SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALPlugInActivator: Illegal char <"> at index 0: "C:\Program Files\Java\jdk1.8.0_202\jre\bin\server
java.nio.file.InvalidPathException: Illegal char <"> at index 0: "C:\Program    Files\Java\jdk1.8.0_202\jre\bin\server
    at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
    at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
    at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
    at java.nio.file.Paths.get(Paths.java:84)
    at org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller.findFolderInPathEnvironment(GDALDistributionInstaller.java:213)
    at org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller.processInstalledWindowsDistribution(GDALDistributionInstaller.java:179)
    at org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller.install(GDALDistributionInstaller.java:67)
    at org.esa.s2tbx.dataio.gdal.activator.GDALPlugInActivator.start(GDALPlugInActivator.java:22)
    at org.esa.snap.runtime.Engine.informActivators(Engine.java:222)
    at org.esa.snap.runtime.Engine.lambda$start$0(Engine.java:121)
    at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
    at org.esa.snap.runtime.Engine.start(Engine.java:121)
    at org.esa.snap.runtime.Engine.start(Engine.java:90)

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters

Is this probably related to the memory error? @marpet @ABraun

#9

InvalidPathException is probably not causing the memory issue. It just means that the inclusion of GDAL will not work.
This error shows that you have specified a different JRE and not the one which is shipped with SNAP. Should still work, but it seems there is an issue with the path. Maybe you have should remove " from the path at the beginning and the end.

You read directly the jp2 file. I’m not sure, but I think in this case there should only be one instance of opj_dump.exe.
You can also try to read the whole product. Open the MTD_MSIL1C.xml directly and then access the band ‘B2’.
Maybe this behaves differently.

I now read that you still use SNAP 6. Can you update to SNAP 7? Maybe this can also resolve some issues.

@kraftek, @oana_hogoiu can you have a look at both issues?