Orbit file timeout (March 2021)

My impression is that it is the RESORB download that is not working in GPT. When I process a product from before 20 (+ a few) days ago, it gets the POEORB files and succeeds.

Downloading the files with PyroSAR and having SNAP pick them up works for me, after some confusion because file system permission issues give exactly the same error message as files not being available online.

PyroSAR has a similar logic to SNAP, cycling through different endpoints and trying to get the different products:

wrapped into a function here

Very descriptively, they call their gathering the orbit files to catch them - they seem to be really hard to get a hold of these days. :wink:

Thank you @j08lue - it seems that RESORB files are removed from the GNSS hub when POEORB files are available. The problem I have is that GPT would not download the POEORB files when they exist on GNSS - anyway I did include the workaround using pyroSAR and it works for me too, but I thought everything should work smoothly just using SNAP/GPT…

I should mention that my solution requires Python. I don’t know how to solve this issue manually.

  1. Install pyroSAR in Anaconda (conda install --channel conda-forge pyroSAR)
  2. Run the following script

from pyroSAR.S1 import OSV

osvdir = r’\OrbitFiles\S1’

with OSV(osvdir) as osv:
files = osv.catch(sensor=‘S1B’, osvtype=‘POE’,
start=‘20170101T000000’, stop=‘20210501T000000’) #define dates here
osv.retrieve(files)

The osvdir will automatically create a folder structure that SNAP expects.
image
image

  1. Take your osvdir and copy and paste the 2 folders (POEORB and RESORB) into C:/Users/yourname/.snap/auxdata/Orbits/Sentinel-1
    Note, this may overwrite/delete something, use caution. I didn’t care if some of my orbit files were deleted because I just downloaded all the possible orbit files anyway.

image
:eyes:

Btw, the concept of “clusterfuck” came to my mind for this issue - a complex system with different parts and processes failing in sequence (surprise API migration, unclear responsibility of different data providers, changes in software that fixed some things but not others, complicated by misleading error messages)… Cluster is a keyword here - an operational system like this is not easy to maintain. But the second part is also very apparent - no offense meant to all the skilled and dedicated individuals, just the cluster.

Sentinel-1 orbit file issue solved :
Try this simple script https://github.com/HelloShagun/SentinelOrbit


i use ur script and then this happened,can you tell me why?

Hi, Thanks for pointing it out. Please try on datasets before 31Mar2021.
Refer to this link : http://step.esa.int/auxdata/orbits/Sentinel-1/
If you have limited SLC files, download it manually from here : https://s1qc.asf.alaska.edu/aux_poeorb/

@shagunsnap and @anab instead of this script, try the PyroSAR implementation mentioned earlier, which handles all those different endpoints.

But this does not mean that this issue is resolved. It just means we all found some hacky workaround. Still waiting on automatic orbit file download to work in SNAP again.

@lveci, the Jira ticket is closed a while ago - will you push a new release with the changes? And will SNAP be able to download RESORB files or only POEORB, as it seems in the fix you implemented: https://github.com/senbox-org/s1tbx/commit/f935b0883de5cb0d3b6c4220be35c8b7eb588313

Hello.

I still have the same problem even though I have manually placed the POEORB files in their corresponding folder.

Am I doing something wrong that I don’t know about?


I have used both https://scihub.copernicus.eu/gnss/#/home and the source https://s1qc.asf.alaska.edu/aux_poeorb/ provided by @shagunsnap

[2] Folder: 20210405
/media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//slaves/20210405
[’/media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//slaves/20210405/S1A_IW_SLC__1SDV_20210405T122554_20210405T122621_037315_046580_2482.zip’]
FILE(s) : /media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//slaves/20210405/S1A_IW_SLC__1SDV_20210405T122554_20210405T122621_037315_046580_2482.zip
[’/home/ignacio/snap/bin/gpt’, ‘/media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//graphs/splitgraph2run.xml’, ‘-c’, ‘10G’, ‘-q’, ‘12’]
SNAP STDOUT:INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 2.3.3 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 2.3.3 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 2.3.3 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 05-APR-2021 12:24:52.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2021-04-01T00:00:000Z TO 2021-04-31T24:00:000Z]
OpenSearch: 0 total results on 1 pages.
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2021-03-01T00:00:000Z TO 2021-03-31T24:00:000Z]
OpenSearch: 0 total results on 1 pages.
No valid orbit file found for 05-APR-2021 12:24:52.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04
done.

Error: [NodeId: Apply-Orbit-File] No valid orbit file found for 05-APR-2021 12:24:52.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04

[2] Finished process in 89.3186290264 seconds.

The screen image is difficult to read, but seems to show “Carpeta Personal/.snap/…”. Please you use the terminal and check that
ls /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04 shows the files.

yes, the files are in the folder, it seems that snap2stamps is not able to read them

From a terminal you can cut and paste as text. Images are not helpful when searching for text in forum posts. Next step is to check file and directory permissions (use ls -ld for each directory starting a ~/.snap, then ls -l *.EOF in the 2012/04 directory) and verify that the downloads are not corrupt. You can try file *.EOF, as in:

$ file S1B_OPER_AUX_POEORB_OPOD_20210501T111617_V20210410T225942_20210412T005942.EOF
S1B_OPER_AUX_POEORB_OPOD_20210501T111617_V20210410T225942_20210412T005942.EOF: XML 1.0 document, ASCII text

If the problem file has different contents (NASA Ocean Color downloads produce html format files when there is a problem with the login). Some sites provide checksums and sizes for the files they distribute.

Hello, thank you very much for your advice.

So far I don’t see anything strange.
I copy the information from my files:

(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/
drwxr-xr-x 3 root root 4096 jul  6  2020 /home/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/
drwxr-xr-x 49 ignacio ignacio 12288 jun 25 17:44 /home/ignacio/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/
drwxr-xr-x 13 ignacio ignacio 4096 ene 27 17:51 /home/ignacio/.snap/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/auxdata/
drwxr-xr-x 13 ignacio ignacio 4096 jun 28 09:49 /home/ignacio/.snap/auxdata/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/auxdata/Orbits/
drwxrwxr-x 3 ignacio ignacio 4096 jul  7  2020 /home/ignacio/.snap/auxdata/Orbits/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/
drwxrwxr-x 4 ignacio ignacio 4096 dic  7  2020 /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/
drwxrwxr-x 4 ignacio ignacio 4096 jul  7  2020 /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/
drwxrwxr-x 10 ignacio ignacio 4096 may 17 13:14 /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/
drwxrwxr-x 7 ignacio ignacio 4096 may 17 13:23 /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/
(base) ignacio@ignacio-G5-5500:~$ ls -ld /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04/
drwxrwxr-x 2 ignacio ignacio 4096 jun 23 18:04 /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04
ls -l /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04/
total 38772
-rw-rw-r-- 1 ignacio ignacio 4409605 jun 23 17:28 S1A_OPER_AUX_POEORB_OPOD_20210404T121856_V20210314T225942_20210316T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409602 jun 23 18:01 S1A_OPER_AUX_POEORB_OPOD_20210405T121944_V20210315T225942_20210317T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409615 jun 23 17:05 S1A_OPER_AUX_POEORB_OPOD_20210417T121642_V20210327T225942_20210329T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409730 jun 23 17:50 S1A_OPER_AUX_POEORB_OPOD_20210423T121738_V20210402T225942_20210404T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409493 jun 23 18:01 S1A_OPER_AUX_POEORB_OPOD_20210424T121724_V20210403T225942_20210405T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409558 jun 23 18:01 S1A_OPER_AUX_POEORB_OPOD_20210426T121637_V20210405T225942_20210407T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409609 jun 23 17:05 S1A_OPER_AUX_POEORB_OPOD_20210429T121756_V20210408T225942_20210410T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409547 jun 23 17:48 S1A_OPER_AUX_POEORB_OPOD_20210508T121724_V20210417T225942_20210419T005942.EOF
-rw-rw-r-- 1 ignacio ignacio 4409549 jun 23 17:49 S1A_OPER_AUX_POEORB_OPOD_20210520T121744_V20210429T225942_20210501T005942.EOF
(base) ignacio@ignacio-G5-5500:~$ file /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04/S1A_OPER_AUX_POEORB_OPOD_20210405T121944_V20210315T225942_20210317T005942.EOF
/home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04/S1A_OPER_AUX_POEORB_OPOD_20210405T121944_V20210315T225942_20210317T005942.EOF: XML 1.0 document, ASCII text

The files types, sizes, and permissions look OK. Copernicus Sentinels POD Service File Format Specification gives details of the file name format for .EOF files. The format uses the underscore (_) to separate fields. For your files:

Option_V – Applicability   Time   Period   <Option_V   suffix>   = _VyyyymmddThhmmss_YYYYMMDDTHHMMSS where:
* yyyymmddThhmmss is the Validity Start Time
* YYYYMMDDTHHMMSSis the Validity Stop Time

You should check these time periods against the error message to make sure you have a file the is valid for the required time.

1 Like

Thank you.
I was effectively selecting the wrong download file.

Thank you snap development team for solving this problem

The summary of this topic is
Orbit file problem is resolved by updating snap s1tbx to version 8.0.4, the default you get when you download snap is 8.0.0 so please update.
For GUI users you can check the version by Tools>plugins and check the installed sentinel 1 toolbox kit module.
For non-GUI users check the version by snap --nosplash --nogui --modules --list --refresh
If update is available update by snap --nosplash --nogui --modules --update-all
(here snap is the location of snap in bin)

Another reason snap might not download the orbit file is that you are working behind a proxy, so you need to setup the proxy by going in snap>bin>gpt.vmoptions and setting the following

-Dhttp.proxyHost=http.proxy.ip
-Dhttp.proxyPort=NNNN
-Dhttps.proxyHost=https.proxy.ip
-Dhttps.proxyPort=MMMM

where http(s).proxy.ip is the IP address of your proxy server and NNNN, MMMM their respective ports.

Check your system set up to discover proxy settings, i.e. via:

echo $http_proxy

credits Setting proxy config with command line - #5 by glemoine

2 Likes

Hello everyone, went through the whole topic feed and understood that this issue is fixed in 8.0.4.
Well, to be sure I reinstalled SNAP with all the latest updates (on Windows 10 WKS).
Unfurtunately I still get the same error: can’t find orbit files in the https://scihub.copernicus.eu/gnss
Can anyone please explain what is the problem with the snappy and POD Hub?@lveci

def apply_orbit(source):
    parameters = HashMap()
    parameters.put('orbitType', 'Sentinel Precise (Auto Download)')
    parameters.put('polyDegree', 3)
    output = GPF.createProduct('Apply-Orbit-File', parameters, source)
    return output

I have the exact same issue as AndrejsZ. I tried to update the modules and even attempted to use the ‘updates=0’ workaround, but neither of them worked.
Hopefully this issue only occurred because of this maintenance:
https://scihub.copernicus.eu/news/News00940

I solved the issue by changing the processing steps in my code, however, I also had an issue when looking for resorb but preorb was already available so to search for both probably is the best option anyway.