Huge execution time difference between graph processing and "manual" processing

qglaude · January 23, 2018, 4:19pm

Dear collegues,

For reasons of ease, I like to create graphs in SNAP. I can focus on other stuffs when my computer is processing background. However, I begin to feel that graph processing is longer that it should be (compared to manually clicking on the needed modules).

To be sure, I used the long graph below and compared the time with using 5 times the second graph + merging.

My results show that it takes 43 minutes for the long graph and 14.4 minutes for 5 times the small graph + manually merging. It means that it takes almost 3 times longer.

To be honest, I really don’t understand the reason for that.

In case of investigation, here are my specs :

Intel i9 7900X
Samsung EVO 960 SSD 1To M.2
64 Go RAM 2400MHz DDR4

I emphasize that my swap is off and I had no memory limitations (peak at 35 go consumed). Also, I run on Linux Mint 18.3 Cinnamon.

mengdahl · January 23, 2018, 4:28pm

I’m not that surprised, large graphs consume more memory and I suspect that the tile cache management becomes inefficient with so many input files being accessed at the same time. Perhaps @forman or @marpet can comment.

If you would script your processing and run it via the command-line you would not need to do much manual work…

qglaude · January 24, 2018, 8:00am

Thank you for your answer.

I understand the memory consumption but I still have plenty of RAM so why not using it?

I never found a utility of using command-line processing. Now i have one. I’ll give it a try.

marpet · February 4, 2018, 2:24pm

If you want to process such an amount it is good to change the tile cache size. It is separate from the overall memory SNAP can use. For the desktop application, you can change the cache in the options panel (Tools / Options in the menu).

If you use gpt from the command line you can use the -c option to configure cache size.
A good value is ~75% of the overall memory for SNAP/gpt. For gpt you can change this in the gpt.vmoptions file. Change the Value after -Xmx in the last line.

qglaude · February 5, 2018, 8:08am

Thanks a lot. It works like a charm now.

For some reasons the Cache size for SNAP desktop was 1024Mo (so i changed it) but the cache in the gpt.vmoptions was already 43Go. it would be nice if SNAP could “guess” how much cache it can use, juste like gpt does.

It’s not a big deal but in the end of the process, I have a problem with the “garbage collection”, saying “GC overhead limit exceeded”. To clear my memory, i have to close snap and open it again.