Orbit file timeout (March 2021)

nirdesh · April 24, 2021, 6:59pm

snap --nosplash --nogui --modules --update-all
snap is located in bin folder of snap

felipemp · April 28, 2021, 5:19pm

I’m still with problems on Linux (Ubuntu 20.04). I have the SNAP version 8.0.3 and I’m trying to run the apply orbit file function on gpt but I’m getting this message:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.2.1 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Connection timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Connection timed out (Connection timed out)
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Connection timed out (Connection timed out)
Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Connection timed out (Connection timed out) due to Connection timed out (Connection timed out)
 done.

Error: [NodeId: Apply-Orbit-File] Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Connection timed out (Connection timed out) due to Connection timed out (Connection timed out)

How can I solve this issue?

s.savastano · April 30, 2021, 10:13am

Hi @florian.beyer,
have you solved this or do you have some hint for me? I think we are facing the same issue: “Apply orbit file” works on SNAP GUI but it gives problem with snappy.
Thank you in advance.

Salvatore S.

florian.beyer · April 30, 2021, 10:38am

Hi @s.savastano,

I’ve found a work around, but I did not have a solution for the actual problem.
My workaround, however, doesn’t work for you except you are working for a federal german organisation
In my case I’m using already preprocessed S1 data on CodeDE (https://code-de.org/de/) within a premium plus account.

My second workaround is now, that I wait for an update, which is solving this issue for snappy.
Thats it… Sorry.

Calebdb · May 3, 2021, 9:59am

Hi, I have the same issue here. It works for the lasted version using the SNAP GUI but not for the gpt process using command line. Something that could we do or just wait for future update?

jcoca · May 3, 2021, 3:20pm

Hello

We can download manually the orbit data from https://scihub.copernicus.eu/gnss/, however we are experiencing problems with sentinel 1 orbits files automatic download.

We are processing in near real time, so we ask por AUX_RESORB files (‘Sentinel Restituted (Auto Download)’) when calling Apply-Orbit-File gpt operator. The automatic download is unsuccessful, please see one example of the output of the operator is:

b"INFO: org.esa.snap.python.gpf.PyOperatorSpi: Python operator ‘py_sambuca_snap_op’ registered (Python module: ‘sambuca_snap_op’, class: ‘sambuca_snap_op’, root: ‘/home/seasnet/.snap/system/modules/org-esa-sen2coral-sen2coral-inversion.jar’)\nINFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters\nINFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.4 found on system. JNI driver will be used.\nINFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.4 set to be used by SNAP.\nINFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.\nINFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.4 set to be used by SNAP.\nINFO: org.hsqldb.persist.Logger: dataFileCache open start\nWARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 30-APR-2021 19:04:05.000000\nOrbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/\nand placed in /home/seasnet/.snap/auxdata/Orbits/Sentinel-1/RESORB/S1A/2021/04\n\nError: [NodeId: someNodeId] No valid orbit file found for 30-APR-2021 19:04:05.000000\nOrbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/\nand placed in /home/seasnet/.snap/auxdata/Orbits/Sentinel-1/RESORB/S1A/2021/04\n"
b’Executing processing graph\nOpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_POEORB AND beginposition:[2021-04-01T00:00:000Z TO 2021-04-31T24:00:000Z]\nOpenSearch: 11 total results on 1 pages.\nOpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_POEORB AND beginposition:[2021-04-01T00:00:000Z TO 2021-04-31T24:00:000Z]\nOpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_POEORB AND beginposition:[2021-05-01T00:00:000Z TO 2021-05-31T24:00:000Z]\nOpenSearch: 0 total results on 1 pages.\nNo valid orbit file found for 30-APR-2021 19:04:05.000000\nOrbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/\nand placed in /home/seasnet/.snap/auxdata/Orbits/Sentinel-1/RESORB/S1A/2021/04\n done.\n’

Although the Restituted are requested, it seems that it is looking for Precise

We were able of download for few days the data using the sentinelsat python module (we use this module to download Sentinel 1 and 2 data). However last week, the orbit file download started to fail. The sentinelsat query, get the desired set of products sentinel 1 orbit files, but fails when try to download throughout uuid:

ConnectionError: HTTPSConnectionPool(host=‘dhusfeed.dhus.onda-dias.net’, port=443): Max retries exceeded with url: /gnss/odata/v1/Products(‘3f838f51-38ed-42b3-bf28-adb4e2a6badb’)/$value (Caused by NewConnectionError(’<urllib3.connection.VerifiedHTTPSConnection object at 0x7f16254ca2e0>: Failed to establish a new connection: [Errno -2] Name or service not known’))

In the information get by the query process. In the link appears https://dhusfeed.dhus.onda-dias.net/gnss instead of https://scihub.copernicus.eu/gnss/. Could be this the problem ? We are not able to access the first web adress. The link that shows this query do not correspond to the Download URL used in https://scihub.copernicus.eu/gnss/. Please see an example of the information retrieved by sentinelsat query:

{‘title’: ‘S1A_OPER_AUX_RESORB_OPOD_20210501T060101_V20210501T013304_20210501T045034’,
‘link’: “https://dhusfeed.dhus.onda-dias.net/gnss/odata/v1/Products(‘3f838f51-38ed-42b3-bf28-adb4e2a6badb’)/$value”,
‘link_alternative’: “https://dhusfeed.dhus.onda-dias.net/gnss/odata/v1/Products(‘3f838f51-38ed-42b3-bf28-adb4e2a6badb’)/”,
‘link_icon’: “https://dhusfeed.dhus.onda-dias.net/gnss/odata/v1/Products(‘3f838f51-38ed-42b3-bf28-adb4e2a6badb’)/Products(‘Quicklook’)/$value”,
‘summary’: ‘Date: 2021-05-01T01:33:04Z, Instrument: , Satellite: Sentinel-1, Size: 546.84 KB’,
‘ondemand’: ‘false’,
‘generationdate’: datetime.datetime(2021, 5, 1, 6, 1, 1),
‘beginposition’: datetime.datetime(2021, 5, 1, 1, 33, 4),
‘endposition’: datetime.datetime(2021, 5, 1, 4, 50, 34),
‘ingestiondate’: datetime.datetime(2021, 5, 1, 6, 10, 10, 182000),
‘format’: ‘EOF’,
‘size’: ‘546.84 KB’,
‘platformname’: ‘Sentinel-1’,
‘platformshortname’: ‘S1’,
‘platformnumber’: ‘A’,
‘platformserialidentifier’: ‘1A’,
‘filename’: ‘S1A_OPER_AUX_RESORB_OPOD_20210501T060101_V20210501T013304_20210501T045034.EOF’,
‘producttype’: ‘AUX_RESORB’,
‘filedescription’: ‘NRT POD Restituted Orbit File’,
‘fileclass’: ‘OPER’,
‘creator’: ‘OPOD’,
‘creatorversion’: ‘1.10.3’,
‘identifier’: ‘S1A_OPER_AUX_RESORB_OPOD_20210501T060101_V20210501T013304_20210501T045034’,
‘uuid’: ‘3f838f51-38ed-42b3-bf28-adb4e2a6badb’}

The SNAP sentinel 1 toolbox we are using is up to date (8.0.3).

Regards
Josep

Rutger · May 4, 2021, 9:58am

Hey jcoca,

I experienced the same issue. The information returned from the query appears to be incorrect, resulting in the “Max retries exceeded” you mentioned when trying to access the provided link. This is the case for both the main and alternative link. As a workaround I ended up just replacing the domain with “https://scihub.copernicus.eu”, it’s a bit hacky, but it made it work (for now ).

Regards,
Rutger

jcoca · May 4, 2021, 3:49pm

Hi

I have decided to use your workaround. It is a little bit weird what is hapenning with the acces to orbit files.

Thank you very much
Josep

florian.beyer · May 5, 2021, 6:37am

@Rutger @jcoca

Could you please describe shortly where you changed the url?

johntruckenbrodt · May 5, 2021, 6:51am

Hi. The return of a GNSS query contains links to ONDA for individual OSV files as well as for additional pages if not all results can be listed on one page. Here the commit of pyroSAR that fixed it: bfd235b.

Rutger · May 5, 2021, 9:34am

@florian.beyer

I ended up just generating a new url from scratch, the response you get (containing the wrong url) also gives you the uuid identifying the product you want. That’s sufficient information by itself, so I ended up adding something like this to my Python code:

url = 'https://scihub.copernicus.eu/gnss/'
if override_url:
    dl_url = f"{url}odata/v1/Products('{uuid}')/$value"

That dl_url is then downloaded with the requests library, resulting in a valid orbit-file.

The replace approach as shown above by @johntruckenbrodt is even better, since you don’t have to make any assumptions about the remainder of the url, just replace the domain part.

j08lue · May 8, 2021, 11:53am

Regarding the automatic download of orbit files in SNAP, which @jcoca mentioned: Will there be a fix in SNAP to prevent the URL redirect, or is this up to the API provider, the Copernicus Open Access Hub, to fix? Would be really great if the two parts were already talking to each other and working on a solution…?

I wrote to the Copernicus EO Support (eosupport@copernicus.esa.int) about this last week. I hope they respond soon. Maybe we can avoid those workarounds in client software.

j08lue · May 11, 2021, 2:45pm

Copernicus EO Support says the issue is fixed on their side:

We have been informed by the technical support team that the error has been rectified.

You should now be able to run your script correctly.

SCI Team
Copernicus Services Coordinated Interface ESA service operated by Serco
Mail: EOSupport@copernicus.esa.int
Portal: https://Spacedata.Copernicus.eu/

I have not verified that yet.

j08lue · May 12, 2021, 10:01pm

The issue with SNAP is still the same:

INFO: org.hsqldb.persist.Logger: dataFileCache open start
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: No valid orbit file found for 04-MAY-2021 05:47:34.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /work/?/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/05
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2021-05-01T00:00:000Z TO 2021-05-31T24:00:000Z]
OpenSearch: 171 total results on 2 pages.
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2021-05-01T00:00:000Z TO 2021-05-31T24:00:000Z]
OpenSearch: https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1 AND platformnumber:A AND producttype:AUX_RESORB AND beginposition:[2021-05-01T00:00:000Z TO 2021-05-31T24:00:000Z]
No valid orbit file found for 04-MAY-2021 05:47:34.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /work/?/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/05
 done.

Error: [NodeId: Apply-Orbit-File] No valid orbit file found for 04-MAY-2021 05:47:34.000000
Orbit files may be downloaded from https://qc.sentinel1.eo.esa.int/ or http://aux.sentinel1.eo.esa.int/
and placed in /work/?/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/05

It seems like the AUX data for my product is not available on the GNSS server. When I query for it specifically (timestamp is 04-MAY-2021 05:47:34.000000), there is no matching result for that exact time:

https://scihub.copernicus.eu/gnss/search?q=platformname:Sentinel-1%20AND%20platformnumber:A%20AND%20producttype:AUX_RESORB%20AND%20beginposition:[2021-05-04T05:00:000Z%20TO%202021-05-04T06:00:000Z]

Are we sure that the implementation of the GNSS endpoint you made here, @lveci, works? https://senbox.atlassian.net/browse/SITBX-843? Maybe a wrong time stamp is used for querying or something? I see the above kind of issue on more than a hundred products by now since the beginning of the month.

@jcoca, do you have a workaround for this, perhaps downloading the GNSS data with Sentinelsat?

zahra0729 · May 15, 2021, 11:41am

I have this problem too, what is the best solution for resolving it?@ABraun

ABraun · May 15, 2021, 1:04pm

Please update to S1TBX 8.0.3

zahra0729 · May 15, 2021, 6:49pm

Thanks for your kind response…

adlauw · May 17, 2021, 2:18pm

Hello !

I still have the same error even after updating S1TBX 8.0.3…

Error: [NodeId: Apply-Orbit-File] No valid orbit file found for 22-APR-2021 05:49:43.000000
Orbit files may be downloaded from qc.sentinel1.eo.esa.int/ or aux.sentinel1.eo.esa.int/
and placed in /home/appuser/.snap/auxdata/Orbits/Sentinel-1/POEORB/S1A/2021/04

Snap has been installed in a Docker using :

RUN wget https://download.esa.int/step/snap/8.0/installers/esa-snap_sentinel_unix_8_0.sh &&
chmod +x esa-snap_sentinel_unix_8_0.sh &&
./esa-snap_sentinel_unix_8_0.sh -q &&
ln -s /usr/local/snap/bin/gpt /usr/bin/gpt &&
sed -i -e ‘s/-Xmx1G/-Xmx16G/g’ /usr/local/snap/bin/gpt.vmoptions
RUN snap --nosplash --nogui --modules --update-all

Does anyone have a workaround for this ?

Many thanks for your help,

Adrien Lauwers

zahra0729 · May 19, 2021, 8:53am

sorry dear ABraun, in snap v_6 can we update to S1TBX 8.0.3?

ABraun · May 19, 2021, 9:05am

no, versions (e.g. 6, 7 or 8) have to be downloaded and installed. Only smaller updates within these (e.g. 8.0.1 to 8.0.3) can be performed from within SNAP.