[SNAP 7.0]: Error reading sentinel-2 data from AWS

Hi, I would like to load sentinel-2 data in SNAP 7.0 from the “sentinel-s2-l2a” bucket at AWS.

I think I have properly configured the VFS settings as the SNAP user help advices. But I get an “503: Forbbiden error” when I try to access to the VFS in order to load a S2 product.

I think SNAP does not include the “–request-payer=requester” parameter or any other necessary setting. Any idea?

I successfully read this bucket with the AWS CLI app.

Thanks in advance.

Ummmm, now I get other error: “301: Moved Pertmanently”:

image

My settings:

Hi, can you try with
Address: https://s3.eu-central-1.amazonaws.com/
(in Remote File Repository Configurations)
?

Thanks for your advice!!!, but I get “403: Forbidden” error :sleepy:

image

UPDATE:
I tried https://sentinel-s2-l2a.s3.eu-central-1.amazonaws.com/ too, same error.

You don’t need the last property you added.
Like this it works:

Also make sure to click on Apply changes and restart SNAP, check your changes are there, and after try to access the VFS in Open Product.

Same “503: Forbbiden error”, my AWS Account has the “AmazonS3ReadOnlyAccess” policy linked, but I tested the “AmazonS3FullAccess” with equal result.

Hi,
503 Status Code is Service Unavailable, … not Forbidden.
Can you attach your last messages.log file from SNAP, please?
You can find it by selecting from the menu Help / Show Log Directory.
Maybe we can find a clue there.
Thank you in advance.

My LOG trace:

SEVERE [org.esa.snap.vfs.ui.file.chooser.VirtualFileSystemView]: Unable to get files. Details: Failed to get the root directories.
java.io.IOException: https://s3.eu-central-1.amazonaws.com/s3bucketl2a/?delimiter=%2F: response code 403: Forbidden
at org.esa.snap.vfs.remote.s3.S3Walker.walk(S3Walker.java:112)
at org.esa.snap.vfs.remote.AbstractRemoteFileSystem.walkDir(AbstractRemoteFileSystem.java:303)
at org.esa.snap.vfs.remote.AbstractRemoteFileSystem.getRootDirectories(AbstractRemoteFileSystem.java:150)
Caused: java.lang.IllegalStateException: Failed to get the root directories.
at org.esa.snap.vfs.remote.AbstractRemoteFileSystem.getRootDirectories(AbstractRemoteFileSystem.java:152)
at org.esa.snap.vfs.ui.file.chooser.VirtualFileSystemHelper.getRootDirectories(VirtualFileSystemHelper.java:51)
[catch] at org.esa.snap.vfs.ui.file.chooser.VirtualFileSystemView.getVirtualFiles(VirtualFileSystemView.java:560)
at org.esa.snap.vfs.ui.file.chooser.VirtualFileSystemView.getFiles(VirtualFileSystemView.java:247)
at javax.swing.plaf.basic.BasicDirectoryModel$LoadFilesThread.run0(Unknown Source)
at javax.swing.plaf.basic.BasicDirectoryModel$LoadFilesThread.run(Unknown Source)

Thanks for your support.

It seems that the access error comes in the response from AWS…so it’s not a matter of error reading from AWS. It’s a matter of error accessing AWS.

With correct credentials for that bucket, it should work (we tested with our account, and the response status code is 200 OK.

1 Like

Thank you very much Oana, I will review my credentials.
Best regards

Hi, I have done some tests using “postman”.

I can not access to the “s3bucktetl2a” that you propose me. Anyway the buckets that I need to load require the “x-amz-request-payer” request header setting. They are some public ones at AWS called “landsat-pds”, “sentinel-s2-l2a”, “sentinel-s2-l1c” and “sentinel-s2-l1c”.

I have to add this request header setting (x-amz-request-payer = requester) in postman in order to get OK results.

From SNAP application, this header settings is never sent (I use Fiddler to sniff the final URL sent). Is this access type really implemented in SNAP? Is this a new feature to develop?

Hello,
For the moment, in the actual implementation, only the headers marked below are sent:

I have created a request for allowing users to add custom AWS headers, that start with “x-amz-”.
https://senbox.atlassian.net/browse/SNAP-1160
Hopefully this will solve your problem (because we cannot reproduce your error with the account we are using…)
Best regards,
Oana

Thank you very much Oana

1 Like

Hi, I did myself some changes in VFS code of snap-engine to support custom headers. If this helps in any manner I will be happy!

Now, this works here!

Hello,
We also did the changes today on 7.x branch, on both snap-engine and snap-desktop.
You can check if you want, to see if it fits you :slightly_smiling_face:
Best regards,
Oana

Hi Oana, thank you very much!!! Wow, what a very fast support!

I have tested this commit and seems ok except for one issue. There are some satellite buckets in Amazon with special URL addresses (The bucket name is self-contained in the URL address).

For example:
- Sentinel:
https://sentinel-s1-l1c.s3.amazonaws.com
https://sentinel-s2-l1c.s3.amazonaws.com
https://sentinel-s2-l2a.s3.amazonaws.com
- Landsat:
https://landsat-pds.s3.amazonaws.com

So the bucket name verification has to be avoided, as I had to do in my own code, in order to successfully read from these AWS imagery repositories.

Do you agree to integrate this change?

My VFS settings:

Best regards,
Alvaro

Hello Alvaro,
Many thanks for your help!
Yes, we integrated this change (proposed by you) on 7.x branch, by checking if the bucket is self-contained and throwing an error only if it’s not self-contained in URL and also not set as a property by the user.
Best regards,
Oana

1 Like

Thank you very much SNAP team!

Now I am going to try load a product from one of these AWS repos.
Best regards
Alvaro

Hello Alvaro,
I am writing just to let you know that the updates for VFS configurations were published yesterday evening on the update center, so you can benefit from them by updating SNAP.
Best regards,
Oana

Hello Oana, thank you very much for the info, congratulations to your whole team, I think SNAP platform is an amazing software. Our project needs to read from AWS buckets therefore I am going to try to fix the S1 IO reader, the local copy of this data is succesfully loaded and I hope the necessary changes be minimal.

1 Like