403 for skywatch-auxdata bucket

a2tirb · May 10, 2021, 7:16pm

Hi,

I get a 403 when I run a graph with gpt:

INFO: java.util.prefs.FileSystemPreferences$1: Created user preferences directory.
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_36_02.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_36_02.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_36_02.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_36_02.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_36_02.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_36_02.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: Server returned HTTP response code: 403 for URL: http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_36_02.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_36_02.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: Server returned HTTP response code: 403 for URL: http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_36_02.zip

I assume it is the terrain correction?

My graph looks like this:

gist.github.com

https://gist.github.com/brwe/2dde01539915b0d2524b798ee112dec0

snap_graph.xml

<graph id="Graph">
  <version>1.0</version>
  <!-- usage example: 
    /usr/local/snap/bin/gpt all-steps.xml -Pinput=PATH_TO_MANIFEST_FILE/manifest.safe -Pvh_result_file_name=PATH_TO_RESULT_FILE/vh.tif -Pvv_result_file_name=PATH_TO_RESULT_FILE/vv.tif 
  -->
  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>${input}</file>

This file has been truncated. show original

This worked fine before but when I tried today I get this error. Any idea what might have caused this or how to fix it?

MHrnciar · May 11, 2021, 8:37am

dealing with the same issue

SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: Server returned HTTP response code: 403 for URL: http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_40_03.zip

marpet · May 11, 2021, 9:16am

Skywatch resides in Canada. They are still sleeping. I think they show up from 12:00 UTC on and can look at the issue.

Right @lveci?

a2tirb · May 11, 2021, 3:44pm

If anyone knows a workaround for this issue I’d also be very grateful for it.

a2tirb · May 12, 2021, 9:27am

Is there maybe a way to use a different download url? Where does the url to http://skywatch-auxdata.s3-us-west-2.amazonaws.com come from? I was hoping I could change that with the settings in etc/snap.auxdata.properties but no luck. Any pointer appreciated.

matthewccasey · May 12, 2021, 11:01am

We’re also getting the same problem, with SNAP then sitting in a loop on the 403 error.

The Skywatch URL returns “AccessDenied”. However, it is available via the aws s3 command line (e.g. “aws s3 ls s3://skywatch-auxdata/dem/SRTM90/tiff/”). This suggests that the S3 bucket has been locked down for public access via HTTP, which SNAP is using for the download. If SNAP only uses the public HTTP endpoint, then I don’t think there is an alternative for the download.

marpet · May 12, 2021, 11:56am

Actually the skywatch bucket is only used as a fallback.

If the data is not found at the URL provided in the snap.auxdata.properties, then https://download.esa.int/step/auxdata/dem/SRTM90/tiff/ is used. Often the same. If this doesn’t work either then the skywatch bucket is used.
So the first problem is that the download from https://download.esa.int/step/auxdata/dem/SRTM90/tiff/ is not working.
But access vie this link is working: https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_36_02.zip

Have you maybe download a lot of data and requested a lot of DEM data in parallel?
Maybe you have been black-listed.
See also this thread: Snap connection to server

@Oana, can you check this?

oana_hogoiu · May 12, 2021, 12:07pm

If the IP used by @matthewccasey accesing this forum is the same one used by SNAP when connecting to STEP server, then this IP is not banned by STEP server.

matthewccasey · May 12, 2021, 12:10pm

Thanks for looking at this. If it helps, to my knowledge, we ran SNAP on the same process only a couple of times yesterday. The first time it certainly worked and now it doesn’t.

In case it is a limit we’ve reached, I’ve re-checked access to https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_36_02.zip and that works via curl. (It also worked yesterday when I first encountered the problem.) So I think that implies we haven’t hit any limits.

I’ve also tried curl for http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_36_02.zip and that returns “Access Denied”, implying that the S3 bucket has been locked down in some way.

matthewccasey · May 12, 2021, 12:10pm

Thanks, yes it will be the same.

oana_hogoiu · May 12, 2021, 12:25pm

In the logs, the only auxdata resources hit by your IP this week were some orbits and egm96 DEM.
There is no record for SRTM90 DEM access.
Therefore, at first sight, not a STEP server access issue.

matthewccasey · May 12, 2021, 12:37pm

Thanks. Odd, because I ran it certainly a couple of times yesterday (if not more) and today. Here’s an example from just now (also ran at least twice earlier today):

And from the same host, curl working fine to the ESA URL (also ran earlier to test):

Am I missing something or is there anything else you’d like me to try?

oana_hogoiu · May 12, 2021, 12:53pm

I see the log entry for the orbit file request (the same one from your attached log):

[12/May/2021:14:31:39 +0200] “GET /auxdata/orbits/Sentinel-1/POEORB/S1A/2018/04/S1A_OPER_AUX_POEORB_OPOD_20180519T120732_V20180428T225942_20180430T005942.EOF.zip HTTP/1.1” 200

but no entries for SRTM90 DEM. Those requests are somehow not sent to STEP server by SNAP.

Also, in your log there are only errors regarding skywatch-auxdata bucket. In case of non-successful requests to STEP server, these errors would also appear in your log…

oana_hogoiu · May 12, 2021, 1:22pm

Just now noticed that the URLs are with download.esa.int, not step.esa.int, this is why they don’t show up in STEP logs.

marpet · May 12, 2021, 1:26pm

Oh yes, you are right. I also heard that Luis is looking into it.

matthewccasey · May 14, 2021, 2:57pm

Just wondered if there was any more information on this - the error is still occurring?

a2tirb · May 18, 2021, 3:16pm

Does anyone have an idea how to work around this? It is still happening to me.

a2tirb · May 18, 2021, 5:41pm

Also, does this only happen to me and @matthewccasey or does anyone else have the problem too?

marpet · May 19, 2021, 6:01am

Hi @a2tirb
no, I have seen this problem too. But, lucky me, I don’t use it for any production. I just tested this functionality.
As I said earlier, @lveci is looking into this. Luis, can you give a status update?.

a2tirb · May 19, 2021, 7:13am

Thanks a lot! I think the exception that occurs when download fails is swallowed too and think that would be good to change in any case. I made a pull request.