Gpt hangs while processing S1 image

Hey community members!

I’ve faced a problem when trying to process S1 image. I want to subset a 1024x1024 window from the original image and then perform Range-Doppler Terrain correction.

To do that I created the following graph.

I’m able to get results when launching this graph using SNAP UI. However, it hangs if I try to run it with gpt.

Any suggestions?

Could it be that you have configured SNAP Desktop and gpt with different memory settings?

Check this I’m getting the error “Cannot construct DataBuffer”, … FAQ entry, please.

It’s very unlikely. I set -Xmx22g in gpt.vmoptions. Here is what ./gpt --diag returned in my case:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: No distribution folder found on Macintosh.
SNAP Release version 6.0
SNAP home: /Applications/snap/bin//..
SNAP debug: null
SNAP log level: null
Java home: /Applications/snap/.install4j/jre.bundle/Contents/Home/jre
Java version: 1.8.0_102
Processors: 16
Max memory: 19.6 GB
Cache size: 0 B
Tile parallelism: 16
Tile size: 512 x 512 pixels

At first, I thought that Cache size: 0 B could be an issue, but apparently it is not. The reason why I think so is that I have the same behavior of gpt when launching it with -c 4096 option.

Moreover, as you can see, I use a subset operator in my graph. It also works in an obscure way (or maybe I don’t see something). Using this operator I’m trying to get a 10580x10580 pixels sub-image and then perform terrain correction on it, but it looks like snap just ignores it. When I run this graph using snap’s UI the processing takes a few minutes, but the result I get has the original resolution which is 26189x16686 in my case.

Attaching XML file for more context.graph.xml (3.6 KB)

I see you select SRTM 3Sec as an input DEM for terrain correction. Have you changed the location of the data as described here? SRTM ZIP-files are corrupted or not found

Yes, I set it to

DEM.srtm3GeoTiffDEM_HTTP = https://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/

According to the instruction taken from here.

does the graph also stop when you select SRTM 1Sec instead?

It does, but with an error Error: The DEM 'SRTM 1Sec' is not supported.

UPD

The actual error is

Terrain-Correction error: no valid output was produced. Please verify the DEM

the correct term should be

<demName>SRTM 1Sec HGT</demName>

I’ve just tried it. No lack. A SRTMGL1.hgt.zip file has been fetched and then I’ve got the same behavior.

I just tested your graph with a random S1 GRD product

Without the adjustment of SRTM 3 source a 1 KB small file is created but never finished which prevents the process to proceed

grafik

I closed SNAP and deleted this file (it occurred two more times because the server request was still running, so I had to wait a bit until it was permanently deleted).

Then I changed the line in the aux file and saved it, and restarted SNAP.
The process now downloads the elevation data from the new source which is not done if the corrupt 1 KB file is already in the folder. So deleting corrupt files is required before the changes are permanently applied.

grafik

Then, the graph finished without problems.

Diasabling cache will hurt performance, but lower memory usage. You have quite a bit of RAM so why not use the cache too?

I’ve checked .snap/auxdata/dem/SRTM 1Sec HGT location. There is one N52E020.SRTMGL1.hgt.zip 3.2M file there. Unfortunately, deleting and fetching this file again didn’t resolve the issue.

Unfortunately, enabling cache didn’t resolve my issue.

please also check the SRTM 3Sec directory for invalid files

The same with SRTM 3Sec .

I cannot reproduce this behavior, sorry.

Are you talking about reproducing from terminal using gpt utility or snap user interface?

both worked for me

After several experiments, I came up with 2 things.

  1. A subset operator from my computational graph works in an obscure way. If I try to subset a small 1024x1024 pixels window, the computation ends and produces the output. However, If I specify target 10980x10980 resolution the processing also produces the result, but it takes almost 5 hours. I have a lot of RAM and such behavior looks very strange for me. It looks like snap didn’t take memory parameters from gpt.vmoptions.
  2. After 5 hours of processing I checked the result and was a bit surprised. The thing is that computational graph performed terrain correction of the whole raster. The result of the subset operator hasn’t been… skipped (I assume this is the reason why it took so long time to produce the output).

I noticed such behavior only when I use gpt from the terminal. If I use snap UI everything works perfectly.

Any ideas what could be wrong here?