Automatic download of orbit files isn't disabling even though I already have them downloaded in the .snap folder

Microwave Toolbox
#1

I downloaded all orbit files (18 gigabytes with ftp from this server aux.sentinel1.eo.esa.int) to paths that look like this

/usr/src # ls /root/.snap/auxdata/Orbits/Sentinel-1/POEORB/2014/08/22 S1A_OPER_AUX_POEORB_OPOD_20140822T122852_V20140731T225944_20140802T005944.EOF

But snap 8 isn’t automatically using these files I downloaded and it still tries to download files from a server that is down or may not be turned on again. I also tried setting an option to not download auxillary files but that didn’t work either

The logs:

# rave at Ryans-MacBook-Pro.local in ~/ms-sar/preprocessing on git:main ✖︎ [17:14:43]
→ docker run snap       
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
INFO: org.esa.s1tbx.commons.io.ImageIOFile: Using FileCacheImageInputStream
INFO: org.hsqldb.persist.Logger: dataFileCache open start
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: ApplyOrbit ignoring error and continuing: org.apache.http.conn.HttpHostConnectException: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
version = 2.36
INFO: java.util.prefs.FileSystemPreferences$1: Created user preferences directory.
.WARNING: org.esa.snap.dataio.netcdf.util.MetadataUtils: Missing configuration property 'snap.dataio.netcdf.metadataElementLimit'. Using default (100).
...10%...Killed

The command I ran in my docker container
/usr/local/snap/bin/gpt -e -Dsnap.gpf.allowAuxdataDownload=false /usr/src/S1_GRD_preprocessing_external_dem.xml -Presolution=30 -Porigin=5 -Pfilter='Refined Lee' -PDEM_FILE_PATH=$3 -Pcrs='GEOGCS["WGS84(DD)", DATUM["WGS84", SPHEROID["WGS84", 6378137.0, 298.257223563]], PRIMEM["Greenwich", 0.0], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH]]' -Pinput=$1 -Poutput=$2

And I’m using the following arguments as inputs to my processing workflow. The year is from 2014 so the orbit file should be sourced (it’s not so recent that it needs to download it from a server).

Input file: "S1A_IW_GRDH_1SSV_20141007T223418_20141007T223443_002729_0030F5_BB2B.SAFE"
Output file: "S1A_IW_GRDH_1SSV_20141007T223418_20141007T223443_002729_0030F5_BB2B.dim"
External DEM: "nasadems/bolivia_NASADEM_NC_s19w063.nc"

How can I disable this and all external downloads? I think it should not be the default to download auxillary files, if the servers tend to go down. Clear instructions on how to download these external files and where to place them in the .snap directory would be really helpful, along with instructions on how to disable external downloads.

The server at qc.sentinel1.eo.esa.int is taking too long to respond.
#2

2 things:

  • snap does only use /year/month/ folder structure for orbit files, so you should omit day.
  • the orbit files in the aux server are aranged by creation datetime (the first timestamp in the filename), while snap wants them in a folder structure that indicates the datetime for when this orbit file is valid (second and third timestamp in the filename: start and end date of valid range).
    So the orbit file that you show (S1A_OPER_AUX_POEORB_OPOD_20140822T122852_V20140731T225944_20140802T005944.EOF) should be in 2014/07/ or 2014/08/, depending on the date of the scene you’re processing. You’ve put it in 2014/08/22 instead, which is the creation date of the file. Snap will not look there, it will only look in the /year/month/ folder of your scene’s aquisition date.
3 Likes
#3

Thanks a bunch for the response, that’s good to know. I’m not familiar with the file name nomenclature of the orbit files, does anyone know which date in this orbit file represents the date the scene was actually acquired?

S1A_OPER_AUX_POEORB_OPOD_20140822T122852_V20140731T225944_20140802T005944.EOF

#4

I had something similar to that in my own data.

In that case, since the validity is both July and August. I just put copies of that orbit file in both of the folders (.../POEORB/S1A/2014/07 and .../POEORB/S1A/2014/08 folder). I did that just to be safe and minimize the chance of errors and it worked for me.

#5

Thanks for the tip @panjibrotoisworo that’s really helpful. I think I see now that I need to duplicate all my orbit files in order to cover all the date and orbit combinations.

For context, I have 4 terabytes of SAR imagery so I can’t handpick the orbit files I need for each file. Ideally, the server would download these files in such a way that they are ready to use with sNAP and don’t need to be duplicated to use with SNAP scenes, since this doubles storage use from 18 Gb to 36 Gb.