How-to for sentinel 2 on AWS

I’m a relatively new user of SNAP and want to search the AWS for images of my specific Area of Interest. I’m trying to use the product library and having difficulty understanding the instructions. is there a ‘beginners guide’ to using the product library?
I know exactly what I want to do, but can’t find out how!

Hello,
First you need to configure the AWS data source credentials, that will be used in Product Library.
You can find some samples in this post:

1 Like

Thanks, I’ve read that post, but I don’t know what credentials I need. I see that you can import them, but where from?
I dont know the name of the region I want (I need images of Wales).
This is why I asked if there was a ‘beginners guide’, and is a fundamental problem with these forums.
As I stated, I am a newcomer to SNAP and simply telling me to ‘do what someone else did’ does not help.

Maybe it is then a good idea to get familiar with the AWS first, don’t you think? :wink:

1 Like

@mattcooper You’ll need to create an AWS account, then create an IAM user, then an access key for that user.

The Sentinel-2 bucket is documented at https://registry.opendata.aws/sentinel-2/, but is not managed by the SNAP developers. You can see there that its region is eu-central-1. That’s where the Amazon data center which stores the data is located, and has no relation to your area of interest.

Also, note that the bucket is marked as “requester pays”. That means that you’ll be billed for the data transfer from Amazon to your computer (assuming you’re not running on an EC2 VM). You can see x-amz-request-payer being set in the screenshot that Oana linked to.

As for the pricing itself, there’s a free tier of 15 GB per month for one year, then you pay $0.0245 per GB as described in https://aws.amazon.com/s3/pricing/.

2 Likes

Thanks, but that sends me down another rabbit hole of confusion. I have an AWS account, but do i sign in as a root user, or as IAM? do i need secret keys? do I have to pay to access this information?
What I want to do seems very simple to me, but is made extremely difficult by the lack of basic information about how to do it.
I opened the product library, added my user credentials for AWS as instructed (after creating an AWS user account), selected the satellite, start and end date, and area of interest. When i press the search button, nothing happens.

This is why I asked if there was a ‘how-to’ guide for this process. I have mentioned this before, how do you expect new users to be able to learn to use tools like SNAP when all the tutorials and so called ‘help’ on the forums is aimed at experienced users?

Thankyou, @lnicola that helps a lot :slight_smile:

For reference, an AWS access key id looks like AKIAJDHLXXXXXXXXXXXX (upper-case letters and digits) and a secret access key is a 40-character string of lower-and uppercase letters, digits, + and /.

@lnicola OK - so i now have the AWS account set up, and have the following in the remote file repositories tab…


still nothing when searching!

@mattcooper Some clarifications on how to setup and use AWS in SNAP Product Library:
1/ the only AWS credentials needed in SNAP Product Library are the accessKeyId and secretAccessKey; their values should be set in Tools -> Options -> General -> Product Library as values for username and password:

No need for region or bucket, the only S2 bucket used in Product Library is sentinel-s2-l1c. (in the post I previously mentioned the configuration was for AWS Virtual File System, which also needs the credentials, but it’s more complex)

2/ Once these credentials are saved (make sure to click on Apply button), when you open Product Library and select the AWS repository, the Account should be automatically detected and displayed. Then insert the other criteria and perform the search:

You mentioned:
“I opened the product library, added my user credentials for AWS as instructed (after creating an AWS user account), selected the satellite, start and end date, and area of interest. When i press the search button, nothing happens”
Please provide some screenshots.

Besides what @oana_hogoiu has said, note you have a typo in the x-amz-request-payer header name.

I’m not sure about the interaction between those settings and the Product Library, but I would still fix it just in case (or remove the settings completely).

Now I understand. Please enter the credentials in Tools -> Options -> General -> Product Library, like I showed in my previous post.

Since it’s me who created the confusion, I will also clarify it:
Tools -> Options -> General -> Remote File Repositories are the settings for VFS.
Tools -> Options -> General -> Product Library are the settings for Product Library.
AWS is supported by both VFS and Product Library (but the whole configuration is not identical).

@oana_hogoiu thankyou, It now searches (and finds images, but no thumbnails?) properly, but doesnt give me an option to download any files!!
any idea why?


I guess i dont need anything in the ‘Remote File Repositories’ tab!

For downloading, select a product, right click on its corresponding row, a small menu should appear, with the Download option:

Regarding the Remote File Repositories, in case you will ever need AWS for VFS, you already have the settings :slight_smile: But this has no effect on Product Library.

And after clicking ‘download’ the following appears!!

This can mean that the account that you used does not have download access for that bucket (just listing rights) or credits to pay for the downloaded product.
HTTP response code 403 means Forbidden. (therefore the download is forbidden for your account)

The AWS permission system is quite intricate, so it’s possible that the IAM user for which you’ve made the key doesn’t have permissions to access S3 (you can also make a key for the root user, IIRC). I don’t remember how the permissions look like by default, but you can check whether your key has been used or not in the IAM console:

Also, note that keys are region-locked. The key in my screenshot would likely not work on eu-central-1 because it’s been defined on us-east-1.

Unfortunately, it’s not easy to debug this. You could install aws-cli, set up the credentials using aws configure, then use it to enumerate a bucket, but it’s relatively involved. Here I’m replacing my key id and secret with invalid values:

I suspect it’s not a billing issue, since you should still qualify for that free tier (and otherwise you’d get a bill at the end of the month, not an access denied error).

Never mind, it doesn’t work for me either:

While aws-cli works fine:

$ aws s3 cp s3://sentinel-s2-l1c/products/2021/2/4/S2A_MSIL1C_20210204T091151_N0209_R050_T34TGQ_20210204T103917/metadata.xml . --request-payer requester
download: s3://sentinel-s2-l1c/products/2021/2/4/S2A_MSIL1C_20210204T091151_N0209_R050_T34TGQ_20210204T103917/metadata.xml to ./metadata.xml

nope, I also enabled ‘s3fullaccess’ for my IAM and it still doesn’t work!!

The problem is that SNAP is using the S3 Website endpoint:

which is not supported for “requester pays” buckets:

Requester Pays buckets do not allow access through the website endpoint. Any request to such a bucket receives a 403 Access Denied response. For more information, see Using Requester Pays buckets for storage transfers and usage.

(source)

And indeed, the error message (which SNAP should display, but currently doesn’t) mentions this:

The solution here would be for SNAP to use the REST API endpoint instead of the Website one.