SNAP takes much more time to process data by using graph builder than the manual operations


#1

Hello everyone,

I am using Graph builder to process a time series of 60 Sentinel-1 images.
My preprocessing step includes the TOPSAR-Split and Apply-orbit-File for each image of the time series.
image
I have got the warning for downloading orbit files:
WARNING: org.esa.s1tbx.io.orbits.sentinel1.StepAuxdataScraper: Unable to connect to http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/S1A/2017/07/: connect timed out
WARNING: org.esa.s1tbx.io.orbits.sentinel1.StepAuxdataScraper: Unable to connect to http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/S1A/2017/08/: connect timed out
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 24-JUL-2017 05:50:39.974600
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/
and placed in /home/le/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2017/07
WARNING: org.esa.s1tbx.io.orbits.sentinel1.StepAuxdataScraper: Unable to connect to http://step.esa.int/auxdata/orbits/Sentinel-1/RESORB/S1A/2017/07/: connect timed out
WARNING: org.esa.s1tbx.io.orbits.sentinel1.StepAuxdataScraper: Unable to connect to http://step.esa.int/auxdata/orbits/Sentinel-1/RESORB/S1A/2017/08/: connect timed out
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: ApplyOrbit ignoring error and continuing: org.esa.snap.core.gpf.OperatorException: No valid orbit file found for 24-JUL-2017 05:50:39.974600
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/
and placed in /home/le/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2017/07
…10%…20%…30%…40%…50%…60%…70%…80%…90% done.
There was no problem with my internet connection. Then it took 10 minutes for downloading the orbit file for each image, and then quickly finished the processing chain. Whilst, it just takes 30 seconds with manual step-by-step. Is it possible to have an option that allows the use of downloaded orbit file for Apply-Orbit-File step? Or does anyone have an solution for this?

My next step is to generate interferogram with the following graph:
image
I used the external DEM option in this processing chain, and I downloaded and merged the SRTM 1sec HGT DEMs from http://step.esa.int/auxdata/dem/SRTMGL1/ that cover my study area. But I also got the problem of execution time. When using GPT command line for my Graph, it took 2 hours to calculate the interferogram for each pairs of images (one swath of each scene). I tried to do manually the same steps of the processing chain and it took totally 20 – 25 minutes. I use a computer with Intel® Xeon® Silver 4110 CPU @ 2.10GHz × 32; 62.6 GB RAM, Ubuntu 18.04.1 LTS.
Is it normal when it takes much more time to process data by using graph builder than the manual operations? How to solve this problem? Thank you for your help!


#2

Please correct me if I’m wrong but I think that if you already have the orb files, it doesn’t download them.

Concerning your overall problem, I had something similar times ago. It was solved by either editing the gpt.vmoptions file. you can see it in ~/snap/bin/

With your 64Go of RAM, you should have something like that :

# Enter one VM parameter per line
# For example, to adjust the maximum memory usage to 512 MB, uncomment the following line:
# -Xmx512m
# To include another file, uncomment the following line:
# -include-options [path to other .vmoption file]
-Xmx43G

You can also also change it using the SNAP-Configuration-Optimiser


#3

Thank you for your answer, @qglaude
I will try to modify the gpt.vmoptions file.

I haven’t downloaded the orbit files yet. But I wonder if we can do the same thing like the DEM case, with the option for using external DEM, then we can use the downloaded DEM. In the Apply-Orbit-File step, we just have two options: “Sentinel Precise (Auto Download)”; and “Sentinel Restituted (Auto Download)”, so I don’t know if we can use downloaded orbit files?
We can see in the SNAP warning that: Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/
and placed in /home/le/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2017/07
So that we can download and store orbit files in directories for each month? How to choose the right orbit file for a specific acquisition?


#4

When you download orbit files, it’s not for a specific acquisition but for a specific date. You have one orbit files for each day for each satellite (i.e. 2 orbits files per day).

If you want precise orbits for one day, you can simply download it from there https://qc.sentinel1.eo.esa.int/aux_poeorb/ and put it in the directory you mentioned. Note that that’s what SNAP is supposed to do. Maybe a firewall or something is blocking SNAP ? I’m not an expert in these things though.

If you still want to manually use precise orbits, you probably know that precise orbits are only available after a time lag of about 2 weeks.

If the last orbit file available, i.e. this one : S1B_OPER_AUX_POEORB_OPOD_20190218T110752_V20190128T225942_20190130T005942

These orbits are not related to the 18th February 2019 but from 28th to 30th January 2019, as you can read from the filename.


#5

Thank you @qglaude for your answer and all the information.
I’ll try to do your suggestion!


#6

Hi @qglaude,

My gpt.vmoptions file is already the same as what you mentioned:

I also changed the cache size for destop application (on Tools / Options) from 1024MB to 44032MB, but I got the error: Cannot construct DataBuffer. Then I changed this value into 33024MB. However, nothing has changed for the execution time when working with gpt command line. It still takes along time to process my graph.

I found your topic that you created about this issue, and your problem has been solved. How did you set your parameters?


#7

here is how I set my parameters my parameters :

As I said, I’m not an expert in the field. I encounter a similar topic times ago so I thought I may help

Now that I think about it. I also ticked this option. Idk if it related