S1 TOPS Coregistration error - Java heap space error?

[JWora]

Dear all,

i am experiencing following problems when processing S1 TOPS Coregistration with ESD.
my S1 dataset located over Iceland (acquisition dates 04.06.17 and 16.06.17 recently downloaded from scihub, we use the external DEM for back geocoding)
First i thought it was the data that was too new (i firstly started a few days ago processing the scenes acquired on 22.06.17 and 10.06.17) and snap couldn’t find precise orbit file at that time yet that’s why error (by looking at the log file)
Then i switched to the older dataset said above and also increase the heap space size -Xms8g (see below) max heap size have been set to -Xmx88g, still i got the below error.
could someone help solving this problem?
i run snap on the remote Linux server (128 gb RAM) from my PC.
Linux OS is Centos 7.0, Snap latest version 5.0 andS1 Toobox version 5.0.5 were used
thank you very much in advance.

---

[username@linux-server ~]$ snap -J-Xms8g
Entire image is outside of SRTM valid area.
Please use another DEM.
Entire image is outside of SRTM valid area.
Please use another DEM.
Entire image is outside of SRTM valid area.
Please use another DEM.
Entire image is outside of SRTM valid area.
Please use another DEM.
Entire image is outside of SRTM valid area.
Please use another DEM.
Entire image is outside of SRTM valid area.
Please use another DEM.
Entire image is outside of SRTM valid area.
Please use another DEM.
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /home/jirathana/.snap/system/var/log/heapdump.hprof …
Java HotSpot™ 64-Bit Server VM warning: record is too large
Java HotSpot™ 64-Bit Server VM warning: record is too large
Java HotSpot™ 64-Bit Server VM warning: record is too large
Heap dump file created [53930262169 bytes in 198.193 secs]
Java heap space
Java heap space
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.StackOverflowError
java.lang.StackOverflowError
java.lang.NullPointerException
java.lang.NullPointerException
Java heap spacejava.lang.StackOverflowError
java.lang.StackOverflowError
java.lang.StackOverflowError
java.lang.StackOverflowError
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException

[ABraun]

did you allow SNAP to use the 128 GB of RAM?

S2 export into tiff - NaturalRGB or Falso color infrared RGB

Hi ertert, please have a look the this thread http://forum.step.esa.int/t/exporting-rgb-image-as-a-geotiff/1071 There I explain how to create a subset if not enough memory is available. Instead of using a smaller region you can also set the step-X/Y values in order to sub-sample the image. Maybe an other option is to increase the memory heap size. In the 'etc' folder of the installation directory you'll find a file named snap.conf. Open it in a text editor. There is the line which starts wit…

[JWora]

@ABraun

i’m not quite sure how many GB of RAM are allowed to use by SNAP on my linux server. And i couldn’t find snap.conf file in the ‘etc’ "folder at all. Is it a kind of ‘hidden’ file in the directory?

one more question: is it possible to coregister S1 (SLC) subset images with this operator> S1 TOPS Coregistration?
Have you got this experience?

thank you for the help
regards,

[ABraun]

i’m not quite sure how many GB of RAM are allowed to use by SNAP on my linux server. And i couldn’t find snap.conf file in the ‘etc’ "folder at all. Is it a kind of ‘hidden’ file in the directory?

You can also accomplish this under Tools > Options. If you leave some RAM for the Operating system it is fine. I’d try 112GB for a start.

one more question: is it possible to coregister S1 (SLC) subset images with this operator> S1 TOPS Coregistration?Have you got this experience?

Yes. Please have a look here:

The Order of DEM Creating Steps

@musicnerd: Change it from this [image] to this [image]

[MIS]

I have the same heap error issue and have tried to apply the fix using the Tools->Options but the VM window cannot be edited in my version of SNAP. I don’t have a snap.conf file either or I would try that.

Any further advice will be appreciated, thanks.

[JWora]

even if i subset the image (reduced 50% of size), i still got the java heap space error.
BTW i haven’t seen the bar used to select the Bursts as well thanks to @ABraun a really good Tipp.

at Tool > options i can’t modify VM parameters as well (they are in grey and faded out), although my admin said that i do have all rights to read and write the file.

thank you for all advices and for further advices! i really would like to get the things solved…

[ABraun]

then the only way is to edit the snap.conf file directly with a text editor.

[funkenbeachin]

Another way to approach the java heap space error: flush memory. Save products, save the session, close SNAP, open SNAP, select File>Reopen Product. Select only the product needed. Alternatively, select File>Session>Open Session, select the session. Some operations, viewing images seems to hold memory, this removes that load.

Also look at creating virtual memory. This is emulated memory on a drive. I have not tested utility of virtual memory by SNAP.

[qglaude]

Also look at creating virtual memory. This is emulated memory on a drive. I have not tested utility of virtual memory by SNAP.

If by that you mean enabling “swapping”, I strongly advise not to use it. Your computer will become extremely slow or even not responding at all.

[funkenbeachin]

My understanding of “swapping” or “disk swapping” is when a multitasking operating system takes an image of memory used for program execution, writes it to a disk, when the program is idle, freeing memory for active program execution. That’s memory management in multitasking environment. Noticeably slow on old hardware, much faster with new, faster bus, faster drive speed.

Virtual memory is similar, where much slower disk is used to increase the amount of virtual memory. However, I have seen virtual memory, showing as “enabled” by default, on a new machine. Still, I have not tested use of virtual memory.

I did watch memory usage by SNAP and solved the java heap space error by restarting SNAP.

[Samira.h]

• Check the system disk usage. If the disk is full, clean up as much as possible the disk, the trash and the SNAP cache.

(check your disk usage with df using command line), for SNAP cash ( you must go to this direction to check this: var/cache to remove tmp files, in the case of your processing, because you use Sentinel1 processing SNAP doesn’t make temporary files in cache folder It makes just for Sentinel2 processing.)

-increase the minimum and maximum java heap size. To do this, modify snap.conf in /usr/local/snapx/etc. Then increase parameters Xms and XMx(respectively minimum and maximum memory size), without exceeding the allocated RAM of the VM.( you can check your memory with free command in command line). Then open with emacs snap.conf file ( you must use it with sudo otherwise it’s just read only and you can’t modify it). you can also increase parameters Xms and Xmx of gpt.vmoptions file in /usr/local/snapx/bin.

[funkenbeachin]

I found that setting the the VM parameter -Xmx , maximum size, to be a critical setting. A value about half the size of total memory has been working. The parameter can be changed in Options>Performance tab, or be edited the configuration file with a text editor. -Xms , minimum size, may be small (256M). Also useful to set the cache size, the medium value of the benchmark tests seems to work better than the smaller, default, value.

[Ikuemonisan]

I ran into java heap space when performing all swath split in SNAP, and I have gone through the contributions on the subject matter but non worked in my favour yet. Can someone simplify the process of resolving heap space error? What do i need to change on the dialogue box (tool -> option https://forum.step.esa.int/uploads/default/original/2X/0/024ae75804f4c48f72e97245c1f5bf3b40b2e0c1.png). I have also increased the SNAP values but no solution yet. I also tried to remove the tmp files but permission denied.

Please I need some help.

[Ikuemonisan]

Hi Samira.h , I tried to remove the tmp file as you suggested but it did not work for me. I got permission denied. Please help.

[ABraun]

Increasing the available memory only has an effect if your PC has this amount of RAM. It only defines how much of your RAM can be used by Java (SNAP in this case).
For example, if you have 8 GB of RAM, increasing the max value to 16 GB has no effect.
m in this context stands or megabytes and the values should ideally be a multiple of 512 (explanation)
You can more flexibly also use G instead: 2G are 2048m, 4G are 4096m, and so on…

Gpt and Snap performance parameters - exhaustive manual needed!

From my experience you can increase the cache size to a value of 75% of the max memory (Xmx) value. Depending on the actual processing the value might vary. In some cases it can be increased up to 90% and sometime a value of 60% is already the limit.

So if you have 16 GB of RAM, you can set it to -Xmx12G
If you only have 8 GB of RAM, you can set it to -Xmx6G

[giangnv]

Dear colleague,

My system is Window10 64bit, corei5, 32GB RAM, ~30GB free space in C drive. How should I optimize configuration of the SNAP Performance (E.g: VM parameter, Cache size, Tile size, Number of Thread).
I have just upgraded SNAP from 6.0 to the version 7.0 and found it is processing slower than before.

The VM parameter line is invisible as image below

[anu]

Increase the size of the cache to 15000 as it is discussed in the forum; tile size as 1024.

[mengdahl]

If you want to change the parameter values you need to run SNAP with admin privileges.

[Shaila]

Hi,

I had the same issue of JAVA HEAP SPACE and i tried to increase the RAM and the specific that you described as I explained in this post SNAP computation on python doesn’t occupy all the RAM available - development / python - STEP Forum (esa.int) but the RAM is not fully occupy by the process and the JAVA HEAP SPACE error persist. What am I doing wrong?

Thanks,
Shaila