Besides what @oana_hogoiu has said, note you have a typo in the x-amz-request-payer header name.
I’m not sure about the interaction between those settings and the Product Library, but I would still fix it just in case (or remove the settings completely).
Now I understand. Please enter the credentials in Tools -> Options -> General -> Product Library, like I showed in my previous post.
Since it’s me who created the confusion, I will also clarify it:
Tools -> Options -> General -> Remote File Repositories are the settings for VFS.
Tools -> Options -> General -> Product Library are the settings for Product Library.
AWS is supported by both VFS and Product Library (but the whole configuration is not identical).
@oana_hogoiu thankyou, It now searches (and finds images, but no thumbnails?) properly, but doesnt give me an option to download any files!!
any idea why?
For downloading, select a product, right click on its corresponding row, a small menu should appear, with the Download option:
Regarding the Remote File Repositories, in case you will ever need AWS for VFS, you already have the settings 
But this has no effect on Product Library.
This can mean that the account that you used does not have download access for that bucket (just listing rights) or credits to pay for the downloaded product.
HTTP response code 403 means Forbidden. (therefore the download is forbidden for your account)
The AWS permission system is quite intricate, so it’s possible that the IAM user for which you’ve made the key doesn’t have permissions to access S3 (you can also make a key for the root user, IIRC). I don’t remember how the permissions look like by default, but you can check whether your key has been used or not in the IAM console:
Also, note that keys are region-locked. The key in my screenshot would likely not work on eu-central-1 because it’s been defined on us-east-1.
Unfortunately, it’s not easy to debug this. You could install aws-cli, set up the credentials using aws configure, then use it to enumerate a bucket, but it’s relatively involved. Here I’m replacing my key id and secret with invalid values:
I suspect it’s not a billing issue, since you should still qualify for that free tier (and otherwise you’d get a bill at the end of the month, not an access denied error).
Never mind, it doesn’t work for me either:
While aws-cli works fine:
$ aws s3 cp s3://sentinel-s2-l1c/products/2021/2/4/S2A_MSIL1C_20210204T091151_N0209_R050_T34TGQ_20210204T103917/metadata.xml . --request-payer requester
download: s3://sentinel-s2-l1c/products/2021/2/4/S2A_MSIL1C_20210204T091151_N0209_R050_T34TGQ_20210204T103917/metadata.xml to ./metadata.xml
nope, I also enabled ‘s3fullaccess’ for my IAM and it still doesn’t work!!
The problem is that SNAP is using the S3 Website endpoint:
which is not supported for “requester pays” buckets:
Requester Pays buckets do not allow access through the website endpoint. Any request to such a bucket receives a
403 Access Deniedresponse. For more information, see Using Requester Pays buckets for storage transfers and usage.
(source)
And indeed, the error message (which SNAP should display, but currently doesn’t) mentions this:
The solution here would be for SNAP to use the REST API endpoint instead of the Website one.
Just wondering if using SciHub from Product Library for downloading S2 products would be an alternative for you. There everything is free, you just need to create an account on
https://scihub.copernicus.eu/dhus/#/home
and then configure it in SNAP for Product Library.
Matt allow me to elaborate a bit on some of yours comments.
Developing and maintaining SNAP and the environment around it requires considerable resources. SNAP is a Free and Open Source Software that is offered to the EO community at best effort. It was developed based on a history of legacy solutions and covers a wide scope (large number of supported missions and features offered). Simply put we can’t cover everything and we rely on the community to contribute and help carry forward the areas where our focus may not have covered all the needs.
For sure we can’t test SNAP against all the environments, infrastructures, operating systems and data types - from time to time something will not be working properly and the community should help by reporting the problem, eventually finding work-arounds and potentially even contributing fixes. This is how active FOSS projects work - out of contributions of everyone.
The other aspect is that we can’t also fully document in a step by step guide all the features. Everyone should see this as an opportunity for contribution, instead of a drawback or limitation. In fact these forums are here for that. To allow the community to exchange experiences, help each other and report problems directly to the core/main developers.
Your case is a very particular one. Using AWS to download and/or consume data is not a scenario for the typical user of SNAP - yes this can be done but it is not something we would actively test. if not else because ESA has no direct responsibility on the provision of data at AWS. You will soon find out that the Sentinel data at AWS is in many cases different (format, etc) from the data ESA officially distributes.
The feature you are referring too exists in SNAP because it was donated/inherited from another activity that the developers followed - I am pretty sure that it is not working anymore because at the time it was coded AWS was still making sentinel data available for free. They have since then change their approach and because we don’t/can’t actively monitor all of these third-party behaviour the feature is now broken.
Two sides. The beginners guide for something so specific is not top priority, you could help the community by generating such a guide yourself so that the next time something exists for someone else. As this being a fundamental problem with these forums, you will to be more specific - not having guides? not helping beginners? Again all content is generated by contributions, if you think something is missing you can offer to help or at least highlight that there is missing information so that someone else tries to help.
Good catch Laurențiu - @oana_hogoiu can you please create an issue to cover this problem. Thanks.
This is the main, point for a beginner, using a cloud solution may be overkill. Specially if all you want is to do a query (discover data) and download a single product/image. You are probably better off with the official distribution channel. Which you can access directly via the official portal or use the product library.
Final points.
@cristianoLopes
Many thanks for your reply, I was not aware that SNAP was not a proprietary ESA product, and I now appreciate even more the help that the community has offered.
Your reply certainly explains the lack of documentation which I was hoping to find!
Yes, I agree that my case is a very particular one, but I think that I am not alone in having such a case. To give a little background, I began my PhD project (looking at forestry and flooding) in March last year as a convert from Oceanography having little or no experience with remote sensing.
For my project i need to determine the NDVI or LAI for a very small region of Wales to use as a variable in a model I’m developing and I’m finding that neither Sentinel or MODIS derived products are produced to the level of precision that needed. To this end I hope to use the raw data to calculate the NDVI myself.
But I digress… the main point of my comments was that (and this applies to most forums of this type, like stackoverflow etc., it is often difficult to find answers to basic questions as many of the members are experienced users who assume that anyone who asks a question, (which is often very specific as they have a specific need) has the same level of experience as they do. So the answers can be very specific.
I once had a question I asked (in which i stated that I was a new user) about Linux in stack overflow answered with a single line of code, no information about how or where to use it!
To return to the present situation, regarding AWS, while i accept that AWS may not be the best route for me to go down for my particular needs, I find the Copernicus data hub and EO browser to be equally frustrating!
I will continue to try and learn how to use SNAP, so may have other questions in the furture!
Thanks Again
@mattcooper if you still want to use AWS, you can use the SNAP product library (or any other product browser) to get a list of the available products in your area of interest and time interval, then download them using e.g. aws-cli. This is especially useful if you want data that’s older than a year (or what the SciHub data retention period is). Unfortunately, AWS stores products in a different format, so you’ll need to find a way to deal with that.
Once you have a list of products, it’s relatively easy to download them automatically.
To this end I hope to use the raw data to calculate the NDVI myself
LAI maybe, but it seems unlikely that you’ll get better NDVI values this way since it’s pretty hard to get an NDVI wrong.
@cristianoLopes This is what I mean
@lnicola Thanks for your reply, but the bit about aws-cli really doesnt mean anything to me.
What is it? is it a Linux command, a python library?
I would question your comment about not getting better NDVI values - please see the attached screenshot:
the bit about aws-cli really doesnt mean anything to me. What is it? is it a Linux command, a python library?
It’s a cross-platform command-line application that (among other things) lets you download files and directories from S3. I’ve shown above an example of using it:
aws s3 cp s3://sentinel-s2-l1c/products/2021/2/4/S2A_MSIL1C_20210204T091151_N0209_R050_T34TGQ_20210204T103917/ . --request-payer requester
This command downloads S2A_MSIL1C_20210204T091151_N0209_R050_T34TGQ_20210204T103917 to the current directory (that product seems incomplete, not sure why). If you have a list of product names like the one above, it’s quite easy to turn them into a list of S3 URLs, then download all of them. Of course, my example is an L1C product, but there’s an L2A bucket on the page I linked before.
I would question your comment about not getting better NDVI values
You’re right, sorry, I forgot about the 300m resolution.
For NDVI, at least, you could even download only the two bands you need to save on transfer costs (and you can even script the NDVI computation so you don’t need to use the UI). I’m not sure what bands the LAI implementation in SNAP needs.
This is what I mean
That’s a bit unfair – I think my posts here are maybe even too verbose, not the other way around. If you say “I have these 100 products I want to download from AWS, what’s the easiest way to do that?”, I can help, but I’m not going to until you’re sure that’s what you want.
"You can use aws-cli" means “it’s possible to do it”, not “go and figure it out yourself”.
Ah, now i see. Thanks but i’ve decide to use the ‘Scientific data hub’ instead, although I’m having trouble finding certain images (I asked another question here: https://forum.step.esa.int/t/using-product-library-to-search/28434)
I’m sorry if I offended you, it wasn’t personal, it just fitted in with my previous answer to @cristianoLopes, I read it as “use aws-cli”, with no indication as to what that is, how to use it, or where to use it. Yes i was frustrated at the time, and I was complaining about people who suggest using things that other people may not know anything about.
In my defence, I’ve been working on my PhD since last March, and I’ve only set foot in my university office once in that period, so learnig new skills and methods of working has ben a bit of a struggle!
Well, this argument is as old as the internet bulletin boards… most of the times the established community is biased against newcomers because they usually want things “handed-over in a silver plate” and don’t show the slightest hint of commitment to helping others helping them (not saying it’s your case). I think you will still find somewhere netetiquette faqs instructing people to when asking a question to provide as much context as possible, to provide what has been tried and results and to ask specific questions --> show commitment to helping others help oneself.
I have to agree with Laurențiu in this case… it is unfair. We live in the information technology era… anything and everything is a few touches/clicks away… if someone mentions aws-cli on a thread that starts with you asking about AWS you can easily figure out they are connected. In ICT world cli means command line interface so that’s a given. Even without this, a quick google search will lead to lots of information about aws-cli including documentation and tutorials.
Coming back to the problem at hand… I see you are getting some help on the other topic already. If you search the internet you will find people that use python with modules like sentinel-sat and others to download data and process it - perhaps this is something you may want to have a look at.
Hi,
You can get Sentinel 2 on AWS information from the ESA and from the USGS. Clients should enroll with the expectation of complimentary admittance to information at one or the other site. Presently ESA conveys Sentinel 2 information in whole areas. These documents are extremely huge, now and then 7+ GB when compacted.