SNAP 6.0 very slow on a linux machine

snap
#1

Hi,
I am pre-processing a Sentinel 1 EW GRD image (S1A_EW_GRDM_1SDH_20141121T095612_20141121T095716_003378_003ED9_6048.SAFE) doing the following steps:

  1. Apply orbit file
  2. Thermal noise removal
  3. Remove GRD Border Noise
  4. Calibration
  5. Multilooking
  6. Terrain correction.
    The multilooking is done over 313 x 313 pixels because I need to bring the SAR EW resolution to 12.5 km.
    The process runs but really slowly. It takes almost 8 hours.
    I tried two different VMs.
    One with these characteristics:
    SUSE Linux Enterprise Server 11 (x86_64)
    16 CPUS and 48G memory
    CPU type: is Intel® Xeon® CPU E5-2690 v3 @ 2.60GHz
    and the runtime was 455 minutes
    The second VM:
    SUSE Linux Enterprise Server 11 (x86_64)
    48 CPUS and 48G memory
    CPU type: is Intel® Xeon® CPU E5-2690 v3 @ 2.60GHz
    and the runtime was 486 minutes.
    During both these periods and the memory and cpu power was entirely adequate , in fact, no threshold was ever reached.
    Any suggestions to speed up SNAP on a Linux Machine?
    The SNAP version I am using is the latest 6.0, but I had the same problem also with the previous version.
    Really many thanks.
    Francesca
#2

Hi @francesca

Are you working on GPT?

M

#3

Both, GPT and GUI…no difference in terms of time performance

#4

Is your DEM-resolution also 12.5km?

#5

In previous experiences with a similar methodology I have seen that the Terrain Correction step is the one increasing processing time. From your post I assume you are working with the complete image? Did you try to include subset?

M

#6

Hi,
I did stop the processing at the multilooking step, without doing the terrain correction and it took hours as well.
We tried the same process on a windows machine with 64 GB RAM and 16 CPUs and it takes 2 minutes.
So I assume the behavior of SNAP in Linux environment a little bit strange. Maybe during the installation is it necessary to do particular steps?

#7

I forced the terrain correction to 12500 m of pixel size.

#8

It is strange that Linux and Windows performance is so radically different - this should not happen. @marpet @lveci any idea what could be going on?

#9

This is indeed strange. We do not treat the one OS differently than the other. The difference I see is that the Linux instances are running as VMs and I suppose Windows not. Maybe this makes the difference? However, the difference is very big.

#10

Hello,
so long time has passed. In the meanwhile some people of the IT team were trying to understand the problem. We tried to force the parallelization setting the snap.parallism=20 in the snap.properites. But the process it is really slow. The IT team believes that SNAP seems to be single thread CPU intensive, whereas our VMware servers are design and optimized for run “parallel processing”.

  • Do you need specific JAVA version to run SNAP
  • Do you need/could use specific start switches or options on SNAP to increase the CPU usage or the parallelism ?
  • Do you have the chance to run SNAP on any VM and verify if you have the same problem in terms of time performance?
  • Did you have the chance to dig a little bit more?
    Really many thanks.
#11

I’m using LinuxMint, and a htop command indicates than all my 10 cores / 20 threads (Intel i9 7900X) are used quite intensively when I use SNAP in most cases (included the terrain correction)

#12

In general SNAP does a parallel computation. It is designed for doing it.
Each band is split in tiles and each tile is processed independently in a separate thread.
But for some specific processing chains, this does not work. For example, if there are interdependencies when computing the resulting bands. And sometimes the whole input scene is necessary for the computation. In this case, the tiling nullified.

To answer your questions:

Do you need specific JAVA version to run SNAP?
SNAP is coming with its own Java VM. So you don’t need an additional one.

Do you need/could use specific start switches or options on SNAP to increase the CPU usage or the parallelism ?
You found the option ( snap.parallelism) already. And when invoking gpt from the command line you can also use the -q option

Do you have the chance to run SNAP on any VM and verify if you have the same problem in terms of time performance?
I used SNAP on VMs in VirtualBox and as OS I used Ubuntu. Also used Proxmox and a Windows OS. I haven’t experienced a significant difference yet.

So, I somehow think that your VMs might have an issue. Are you using other applications on these VMs? How do they perform?

#13

Hi @francesca
Along the comments from the colleagues above, please verify the maximum memory defined for the JVM in both SNAP GUI and GPT depending on the one you use. (standard installation saves those files in $HOME/snap/etc/snap.conf and $HOME/snap/bin/gtp.vmoptions where the respective parameters are -J-XmX and the -Xmx)
For example: In my linux VM with 32GB RAM I set those to 21G as follows: -Xmx21G

In addition, in the snap.properties you can define some other parameters to optimise the processing.
Personally I have defined as follows:
Tile cache size [Mb]
snap.jai.tileCacheSize = 10240
Default tile size in pixels
snap.jai.defaultTileSize = 2048

I hope this helps!

1 Like
#14

In my experience, the bottleneck using SNAP is RAM or Hard Disk, the latter becomes critical when you are stacking a considerable amount of raster files. Are you writing a big amount of data to disk? What kind of hard drive do you use? What is your filesystem? Could you post the /etc/fstab file of your system and point to what partition are you trying to write?

#15

I had a similar issue on a AWS linux machine.
I didn’t manage to speed up the process even though I was using 96 processors.

Changing the defaults snap.jai.tileCacheSize and snap.jai.defaultTileSize (I did x8) solved my issue.
The parameters are in the snap.properties file, in [install_dir]/snap/etc directory.