Orbit file timeout (March 2021)

a2tirb · May 31, 2021, 10:23am

To implement this workaround (placing orbit files in /home/user/.snap/auxdata/Orbits/Sentinel-1/RESORB) I can still get RESORB files from GNSS between Apr 30 and today from scihub but nothing before that.

So I can get all orbit files until end march this year from http://step.esa.int/auxdata/orbits/Sentinel-1/RESORB/ and everything from Apr 30 until today from https://scihub.copernicus.eu/gnss

But what about the orbit files from April?

Does anyone know where I can get them from?

lveci · May 31, 2021, 6:36pm

The RESORBs are only useful until the precise orbits become available.

a2tirb · May 31, 2021, 9:44pm

Yes, I know. I was just hoping that when I hack around issues at least I would not have to deal with two different kind of orbit files.

a2tirb · May 31, 2021, 10:18pm

In case anyone else wants to download all orbits from https://scihub.copernicus.eu/gnss/search here is a very raw script for that: https://gist.github.com/brwe/ff7db758d3f6ea270113cd7fd86680cb

j08lue · June 1, 2021, 7:28am

It seems like the issue could be related to filesystem permissions in Docker containers: When the different fail-over orbit download methods try to write to disk, they fail. And since everything is wrapped in try ... catch all, it comes out as if files were not found online, while the real issue is that they cannot be written.

github.com

senbox-org/s1tbx/blob/f935b0883de5cb0d3b6c4220be35c8b7eb588313/s1tbx-io-ephemeris/src/main/java/org/esa/s1tbx/io/orbits/sentinel1/SentinelPODOrbitFile.java#L95-L97


} catch(Exception e) {
    // try next
}

Since the default behavior is to write the orbit files to user home dir, if that does not exist or the runtime user does not have permissions to write to that for some reason, you get “No orbit file found”.

Digging on…

snasonova · June 1, 2021, 4:17pm

I posted this on a different thread but I solved the problem by downloading all the orbit files:

I am a little late to the party but I found that the python pySAR package is great for downloading precise orbit files. It finds all the files and downloads them into folders with the same structure as snap expects (C:\Users\me.snap\auxdata\Orbits\Sentinel-1).

pyrosar.readthedocs.io

Handling of Orbit State Vector Files — pyroSAR documentation

from pyroSAR.S1 import OSV

osvdir = r’\OrbitFiles\S1’

with OSV(osvdir) as osv:
files = osv.catch(sensor=‘S1B’, osvtype=‘POE’,
start=‘20170101T000000’, stop=‘20210501T000000’)
osv.retrieve(files)

j08lue · June 2, 2021, 7:20am

Yes, you can download externally, but will SNAP pick them up? I have a hard time getting SNAP/GPT to find the files, even though they are just where it expects them.

pmallas · June 2, 2021, 12:43pm

Well, there a couple of things with regard to manual downloads and organizing the files in SNAP that I think I understand. But maybe not.

First, do the files needed to be zipped in order for SNAP to find them? I noticed that the files previously downloaded by SNAP were zipped and not just the plain *.EOF.

Secondly, I think there are three dates in the *.EOF file name - first is the date of file production and the last two are the date range of validity for file. So, in order to get the file into the correct SNAP orbit directory (which is organized by Sat, year, and month) you have to look file name to establish Satellite (very first part of name) and the second date (for year and month).

Am I wrong about any of this?

andrea.melchiorre · June 3, 2021, 8:03am

hello @lveci - it is not clear to me if SNAP 8.0.3 now searches POE orbit files on the GNSS hub. I am still having issues with automatic processing using GPT… this is the log info I get

Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 01-APR-2021 18:13:25.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/andrea/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1B/2021/04
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:B AND producttype:AUX_RESORB AND beginposition:[2021-04-01T00:00:000Z TO 2021-04-31T24:00:000Z]
OpenSearch: 0 total results on 1 pages.
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:B AND producttype:AUX_RESORB AND beginposition:[2021-03-01T00:00:000Z TO 2021-03-31T24:00:000Z]
OpenSearch: 0 total results on 1 pages.
No valid orbit file found for 01-APR-2021 18:13:25.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/andrea/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1B/2021/04

and this causes my script to stop as no orbit files are found. Any idea of possible solutions?
Thank you

j08lue · June 3, 2021, 12:15pm

My impression is that it is the RESORB download that is not working in GPT. When I process a product from before 20 (+ a few) days ago, it gets the POEORB files and succeeds.

Downloading the files with PyroSAR and having SNAP pick them up works for me, after some confusion because file system permission issues give exactly the same error message as files not being available online.

PyroSAR has a similar logic to SNAP, cycling through different endpoints and trying to get the different products:

github.com

johntruckenbrodt/pyroSAR/blob/bfd235ba587c2ef54ed738fcc6116319b38ce189/pyroSAR/S1/auxil.py#L413-L449


def catch(self, sensor, osvtype='POE', start=None, stop=None, url_option=1):
    """
    check a server for files
    Parameters
    ----------
    sensor: str or list
        The S1 mission(s):
         - 'S1A'
         - 'S1B'
         - ['S1A', 'S1B']
    osvtype: {'POE', 'RES'}
        the type of orbit files required
    start: str
        the date to start searching for files in format YYYYmmddTHHMMSS
    stop: str
        the date to stop searching for files in format YYYYmmddTHHMMSS
    url_option: int
        the URL to query for scenes
         - 1: https://scihub.copernicus.eu/gnss

This file has been truncated. show original

wrapped into a function here

github.com

johntruckenbrodt/pyroSAR/blob/5dd1db3f3ecaf141b1a03020279bbce0c0d11ba8/pyroSAR/drivers.py#L1508-L1535


def getOSV(self, osvdir=None, osvType='POE', returnMatch=False, useLocal=True, timeout=20):
    """
    download Orbit State Vector files for the scene
    Parameters
    ----------
    osvdir: str
        the directory of OSV files; subdirectories POEORB and RESORB are created automatically;
        if no directory is defined, the standard SNAP auxdata location is used
    osvType: str or list
        the type of orbit file either 'POE', 'RES' or a list of both;
        if both are selected, the best matching file will be retrieved. I.e., POE if available and RES otherwise
    returnMatch: bool
        return the best matching orbit file?
    useLocal: bool
        use locally existing files and do not search for files online if the right file has been found?
    timeout: int or tuple or None
        the timeout in seconds for downloading OSV files as provided to :func:`requests.get`
    Returns

This file has been truncated. show original

Very descriptively, they call their gathering the orbit files to catch them - they seem to be really hard to get a hold of these days.

andrea.melchiorre · June 3, 2021, 1:09pm

Thank you @j08lue - it seems that RESORB files are removed from the GNSS hub when POEORB files are available. The problem I have is that GPT would not download the POEORB files when they exist on GNSS - anyway I did include the workaround using pyroSAR and it works for me too, but I thought everything should work smoothly just using SNAP/GPT…

snasonova · June 3, 2021, 7:40pm

I should mention that my solution requires Python. I don’t know how to solve this issue manually.

Install pyroSAR in Anaconda (conda install --channel conda-forge pyroSAR)
Run the following script

from pyroSAR.S1 import OSV

osvdir = r’\OrbitFiles\S1’

with OSV(osvdir) as osv:
files = osv.catch(sensor=‘S1B’, osvtype=‘POE’,
start=‘20170101T000000’, stop=‘20210501T000000’) #define dates here
osv.retrieve(files)

The osvdir will automatically create a folder structure that SNAP expects.

Take your osvdir and copy and paste the 2 folders (POEORB and RESORB) into C:/Users/yourname/.snap/auxdata/Orbits/Sentinel-1
Note, this may overwrite/delete something, use caution. I didn’t care if some of my orbit files were deleted because I just downloaded all the possible orbit files anyway.

j08lue · June 12, 2021, 4:32am

j08lue · June 13, 2021, 6:49am

Btw, the concept of “clusterfuck” came to my mind for this issue - a complex system with different parts and processes failing in sequence (surprise API migration, unclear responsibility of different data providers, changes in software that fixed some things but not others, complicated by misleading error messages)… Cluster is a keyword here - an operational system like this is not easy to maintain. But the second part is also very apparent - no offense meant to all the skilled and dedicated individuals, just the cluster.

shagunsnap · June 15, 2021, 3:30pm

Sentinel-1 orbit file issue solved :
Try this simple script https://github.com/HelloShagun/SentinelOrbit

anab · June 17, 2021, 2:28am

i use ur script and then this happened,can you tell me why?

shagunsnap · June 17, 2021, 8:13am

Hi, Thanks for pointing it out. Please try on datasets before 31Mar2021.
Refer to this link : http://step.esa.int/auxdata/orbits/Sentinel-1/
If you have limited SLC files, download it manually from here : https://s1qc.asf.alaska.edu/aux_poeorb/

j08lue · June 17, 2021, 9:15am

@shagunsnap and @anab instead of this script, try the PyroSAR implementation mentioned earlier, which handles all those different endpoints.

But this does not mean that this issue is resolved. It just means we all found some hacky workaround. Still waiting on automatic orbit file download to work in SNAP again.

@lveci, the Jira ticket is closed a while ago - will you push a new release with the changes? And will SNAP be able to download RESORB files or only POEORB, as it seems in the fix you implemented: https://github.com/senbox-org/s1tbx/commit/f935b0883de5cb0d3b6c4220be35c8b7eb588313

nachin6789 · June 24, 2021, 8:11am

Hello.

I still have the same problem even though I have manually placed the POEORB files in their corresponding folder.

Am I doing something wrong that I don’t know about?

I have used both https://scihub.copernicus.eu/gnss/#/home and the source https://s1qc.asf.alaska.edu/aux_poeorb/ provided by @shagunsnap

[2] Folder: 20210405
/media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//slaves/20210405
[’/media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//slaves/20210405/S1A_IW_SLC__1SDV_20210405T122554_20210405T122621_037315_046580_2482.zip’]
FILE(s) : /media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//slaves/20210405/S1A_IW_SLC__1SDV_20210405T122554_20210405T122621_037315_046580_2482.zip
[’/home/ignacio/snap/bin/gpt’, ‘/media/ignacio/Seagate Expansion Drive/ignacio/DF_desc/2020//graphs/splitgraph2run.xml’, ‘-c’, ‘10G’, ‘-q’, ‘12’]
SNAP STDOUT:INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 2.3.3 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 2.3.3 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 2.3.3 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 05-APR-2021 12:24:52.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2021-04-01T00:00:000Z TO 2021-04-31T24:00:000Z]
OpenSearch: 0 total results on 1 pages.
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2021-03-01T00:00:000Z TO 2021-03-31T24:00:000Z]
OpenSearch: 0 total results on 1 pages.
No valid orbit file found for 05-APR-2021 12:24:52.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04
done.

Error: [NodeId: Apply-Orbit-File] No valid orbit file found for 05-APR-2021 12:24:52.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04

[2] Finished process in 89.3186290264 seconds.

gnwiii · June 24, 2021, 11:41am

The screen image is difficult to read, but seems to show “Carpeta Personal/.snap/…”. Please you use the terminal and check that
ls /home/ignacio/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04 shows the files.