Basic error with Snappy (Python beginner)

Dear colleagues,

I do not have experience with Python (just Shell scripting and Matlab…), but started to work with Snappy today in order to enhance the capabilities of my SAR applications.

So, I am facing a BASIC problem to save an ALOS-1 scene after Deskewing it.
Here is the simple script:

import sys
from snappy import ProductIO
from snappy import GPF
from snappy import jpy

file = sys.argv[1]
product = ProductIO.readProduct(file)

HashMap = jpy.get_type('java.util.HashMap')

params_desk = HashMap()
params_desk.put('PdemName', "SRTM 1Sec HGT")
final = GPF.createProduct('ALOS-Deskewing', params_desk, product)

ProductIO.writeProduct(final, "/home/oceano/mestrado/ALOS1/L1.1/20110419/teste_p/snappy_output.dim", "BEAM-DIMAP")

So, when the ‘writeProduct’ is running appears the error message : “RuntimeError: org.esa.snap.core.gpf.OperatorException: Cannot construct DataBuffer

Please, may someone help me?

Thanks!

I am using an 8 cores cpu with 32 GB.

@gusortiz Try changing PdemName to demName, gpt gives all parameters to operators with a ‘P’ prefix but in snappy this isn’t required :slight_smile:

@CiaranEvans and all, I have done the change proposed but still not working.

Probably is a memory error, isn’t?

Using the graphical interface, the optimized configuration (which works pretty well) is:

 -Xmx21504m  -Xms256m -XX:+AggressiveOpts -Xverify:none -Dnetbeans.mainclass=org.esa.snap.main.Main -Dsun.java2d.noddraw=true -Dsun.awt.nopixfmt=true -Dsun.java2d.dpiaware=false 

Cache size: 1024 Mb
Threads: 8

How to apply the same configuration to use with Snappy?

In order to increase the memory snappy can use, you need to edit snappy.ini

The cache size and the thread settings should be taken from the snap.properties files in the etc directory of the SNAP installation folder.
The properties are: snap.jai.tileCacheSize and snap.parallelism

@marpet, I’ve changed the:

  • max memory config to ‘22G’
  • tileCacheSize = 1024
  • defaultTileSize= 100000

And now the new error message is:

SEVERE: org.esa.s1tbx.io.SARReader: org.esa.s1tbx.io.ceos.alos.AlosPalsarProductReader
[input=/home/oceano/mestrado/ALOS1/L1.1/20110419/ALPSRP278756690-L1.1.zip]:
No memory left for cache!

Maybe you leave the tile size unchanged.
100000 would mean that the whole scene is processed in one go. Probably not a good idea.
maybe you can increase it to 1024 which would mean the scene is processed in 1024x2014 pixel tiles.
It might help to lower the value. Maybe just 100. Then only smaller pieces of the product are processed one after the other.

Still the same error… “No memory left for cache!”
When I change the output format to ‘Geotiif’, the error is: “Cannot construct DataBuffer”

Configuration:
snap.properties

  • snap.jai.tileCacheSize = 1024
  • snap.jai.defaultTileSize = 256 (tried from 64 to 1024)
  • snap.parallelism = 8

snappy.ini

  • java_max_mem: 21504M

jpyutil
jvm_maxmem=‘21504M’

All the functions from GPF run OK, apparently. The problem occurs with ProductIO.writeProduct.

GOT IT, @marpet!

Besides jpyconfig.py, snappy.ini and snap.properties, the file __init__.py must also be modified.

Really? What have you changed there?

@marpet, I have changed:
max_mem = '21504M'
at line 213

But this should actually be read from snappy.ini in the 2 following lines.
Maybe this is not working.
However, you finally made it. :+1:

This is not working because everywhere on this forum you explain it wrong:
The line in snappy.ini that reads:
# java_max_mem: 4G
has to be changed to:
java_max_mem: 10G
or whatever you think is a sensible memory limit given your hardware.

The important thing is to remove the hash-symbol (#) at the beginning of the line. Python appears to treat the hash-symbol as indicating a comment and ignores the line.

1 Like

Removing the hash is obvious to me, so I didn’t say it explicitly. But thanks for pointing this out.

1 Like

I question the functionality of the snap.properties file and would really like to understand this:

So there are 3 files in which properties can be set:

  1. /Python34/Lib/snappy/snappy.ini
  2. /Users/.snap/etc/snap.properties
  3. /Program Files/snap/etc/snap.properties
  1. is setting the maximum RAM when running snappy and has no effect on SNAP Desktop. So this one is fine, I understand it and adjusted the value

  2. is always automatically created new when running a process in SNAP Desktop. Even if the file is deleted(!) it will be recreated although with some lines missing and might cause an error in SNAP Desktop. It is not recreated when running a process in snappy. Neither snap.jai.tileCacheSize nor snap.parallelism seem to have any impact on anything I observed (memory usage, cpu 1-4 usage, time of computations)

  3. is basically the same file, but does not get recreated and I think it is ignored by SNAP Desktop and snappy alike

If someone could explain what exactly is going on, I would be really interested to understand it and make some adjustments to increase computation efficiency (although both SNAP Desktop and snappy are currently running without major problems on my computer)

Thanks!

  1. snappy.ini just configures how the JVM is instantiated from Python and does not interfere with SNAP Desktop or gpt.
  2. In this snap.properties user specific properties are stored. So different users can have different settings. It can be deleted and this should not cause any error. Have you seen one? maybe it is not recreated by snappy, because no properties have be changed?
  3. This is the settings file for the SNAP installation. The settings here are overwritten by the snap.properties in (2). So to say these are the default settings. It is probably not recreated because it is in the installation directory and (in general) files should not be deleted from there.

You can get the values of the properties via the following code:

from snappy import EngineConfig
print(‘tileCachSize’, EngineConfig.instance().preferences().get(‘snap.jai.tileCacheSize’, None))

Thank you marpet, that was a helpful answer. Still I am confused about the performance:

  • I found out that processing time on my computer is reduced when reducing(!) the tileCacheSize, although I thought processing should be faster when using more of the RAM? My test process (calibration+deburst) takes 7:30 min for tileCahceSize of 8192 [MB] and is reduced to about 4 min using only 16 [MB], which seems pretty small to me. I’m working on a laptop with SSD dive so maybe the difference between using RAM and using disc isn’t all that much? Still, SNAP Desktop completes the same process in just 2 min and uses nearly 100% CPU while my python script makes use of only 30-40% CPU (all 4 CPUs equally).

  • Parallelism doesn’t seem to have any effect on my computer. All 4 CPUs are used, even if I set parallelism to 1 (which is a good thing, but still) and processing time basically stays the same

@marpet Do you have any ideas about this?

How to let more cores running?