I do not have experience with Python (just Shell scripting and Matlab…), but started to work with Snappy today in order to enhance the capabilities of my SAR applications.
So, I am facing a BASIC problem to save an ALOS-1 scene after Deskewing it.
Here is the simple script:
import sys
from snappy import ProductIO
from snappy import GPF
from snappy import jpy
file = sys.argv[1]
product = ProductIO.readProduct(file)
HashMap = jpy.get_type('java.util.HashMap')
params_desk = HashMap()
params_desk.put('PdemName', "SRTM 1Sec HGT")
final = GPF.createProduct('ALOS-Deskewing', params_desk, product)
ProductIO.writeProduct(final, "/home/oceano/mestrado/ALOS1/L1.1/20110419/teste_p/snappy_output.dim", "BEAM-DIMAP")
So, when the ‘writeProduct’ is running appears the error message : “RuntimeError: org.esa.snap.core.gpf.OperatorException: Cannot construct DataBuffer”
In order to increase the memory snappy can use, you need to edit snappy.ini
The cache size and the thread settings should be taken from the snap.properties files in the etc directory of the SNAP installation folder.
The properties are: snap.jai.tileCacheSize and snap.parallelism
SEVERE: org.esa.s1tbx.io.SARReader: org.esa.s1tbx.io.ceos.alos.AlosPalsarProductReader
[input=/home/oceano/mestrado/ALOS1/L1.1/20110419/ALPSRP278756690-L1.1.zip]:
No memory left for cache!
Maybe you leave the tile size unchanged.
100000 would mean that the whole scene is processed in one go. Probably not a good idea.
maybe you can increase it to 1024 which would mean the scene is processed in 1024x2014 pixel tiles.
It might help to lower the value. Maybe just 100. Then only smaller pieces of the product are processed one after the other.
This is not working because everywhere on this forum you explain it wrong:
The line in snappy.ini that reads:
# java_max_mem: 4G
has to be changed to:
java_max_mem: 10G
or whatever you think is a sensible memory limit given your hardware.
The important thing is to remove the hash-symbol (#) at the beginning of the line. Python appears to treat the hash-symbol as indicating a comment and ignores the line.
I question the functionality of the snap.properties file and would really like to understand this:
So there are 3 files in which properties can be set:
/Python34/Lib/snappy/snappy.ini
/Users/.snap/etc/snap.properties
/Program Files/snap/etc/snap.properties
is setting the maximum RAM when running snappy and has no effect on SNAP Desktop. So this one is fine, I understand it and adjusted the value
is always automatically created new when running a process in SNAP Desktop. Even if the file is deleted(!) it will be recreated although with some lines missing and might cause an error in SNAP Desktop. It is not recreated when running a process in snappy. Neither snap.jai.tileCacheSize nor snap.parallelism seem to have any impact on anything I observed (memory usage, cpu 1-4 usage, time of computations)
is basically the same file, but does not get recreated and I think it is ignored by SNAP Desktop and snappy alike
If someone could explain what exactly is going on, I would be really interested to understand it and make some adjustments to increase computation efficiency (although both SNAP Desktop and snappy are currently running without major problems on my computer)
snappy.ini just configures how the JVM is instantiated from Python and does not interfere with SNAP Desktop or gpt.
In this snap.properties user specific properties are stored. So different users can have different settings. It can be deleted and this should not cause any error. Have you seen one? maybe it is not recreated by snappy, because no properties have be changed?
This is the settings file for the SNAP installation. The settings here are overwritten by the snap.properties in (2). So to say these are the default settings. It is probably not recreated because it is in the installation directory and (in general) files should not be deleted from there.
You can get the values of the properties via the following code:
from snappy import EngineConfig
print(‘tileCachSize’, EngineConfig.instance().preferences().get(‘snap.jai.tileCacheSize’, None))
Thank you marpet, that was a helpful answer. Still I am confused about the performance:
I found out that processing time on my computer is reduced when reducing(!) the tileCacheSize, although I thought processing should be faster when using more of the RAM? My test process (calibration+deburst) takes 7:30 min for tileCahceSize of 8192 [MB] and is reduced to about 4 min using only 16 [MB], which seems pretty small to me. I’m working on a laptop with SSD dive so maybe the difference between using RAM and using disc isn’t all that much? Still, SNAP Desktop completes the same process in just 2 min and uses nearly 100% CPU while my python script makes use of only 30-40% CPU (all 4 CPUs equally).
Parallelism doesn’t seem to have any effect on my computer. All 4 CPUs are used, even if I set parallelism to 1 (which is a good thing, but still) and processing time basically stays the same