Apihub automatic download for S2 tiles

Optical Toolbox
1

There should be a category for the scihub/apihub issues in this forum, but as I am a S2 user, I am posting it in the S2tbx category.

I have written a very simple tool to make programmatic downloads of S2 images. It is available as open source here. It is working very well to download Sentinel-1 and Sentinel-2 products, but I have tried to add an option to download the data tilewise, and it always fails:
The URL I am using to retrieve a tile is, for instance :
https://scihub.copernicus.eu/apihub/odata/v1/Products('03365863-20b3-42b9-bff6-7879bfe87816')/Nodes('S2A_OPER_PRD_MSIL1C_PDMC_20160128T023916_R094_V20160125T111611_20160125T111611.SAFE')/Nodes('GRANULE')/Nodes('S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T31TCJ_N02.01')/$value

It gets the following answer:
<error><code/><message xml:lang="en">An exception occured while creating the stream for node S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T31TCJ_N02.01</message></error>
Do you see anything wrong with the URL I use ?
Thanks in advance !

2

Are you using wget? I noticed that it has issues trying to get stuff directly from nodes. I use the python requests library instead, which seems to work well. Also handy to check whether the response is not HTML or XML, which usually happens when the hub is down (which is not uncommon…).

3

and you are looking at a directory node, so you should use Nodes() instead of $value

4

Thanks for replying on a Saturday Guido !

Yes, I am using wget. But I’ll try the python libraries. Which one are you using, urllib2 ?

I was hoping that $value would bring the whole directory but it doesn’t. I also tried to download the jp2 files but got the same error message.

https://scihub.copernicus.eu/apihub/odata/v1/Products(‘03365863-20b3-42b9-bff6-7879bfe87816’)/Nodes(‘S2A_OPER_PRD_MSIL1C_PDMC_20160128T023916_R094_V20160125T111611_20160125T111611.SAFE’)/Nodes(‘GRANULE’)/Nodes(‘S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T31TCJ_N02.01’)/Nodes(‘IMG_DATA’)/Nodes(‘S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T31TCJ_B01.jp2’)/$value

I also tried to ask the support address (EOSupport@copernicus.eu), and their advice was to ask here !

5

Olivier,

I use the requests python library. I think your URL for the individual granule Band 1 jp2 is correct, but wget will probably return the XML describing the node.

Guido

6

I did not know that one, much simpler than urllib2 ! Thanks.

Still, it is not working:
>>> url "https://scihub.copernicus.eu/dhus/odata/v1/Products('03365863-20b3-42b9-bff6-7879bfe87816')/Nodes('S2A_OPER_PRD_MSIL1C_PDMC_20160128T023916_R094_V20160125T111611_20160125T111611.SAFE')/Nodes('GRANULE')/Nodes('S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T31TCJ_N02.01')/Nodes('IMG_DATA')/Nodes('S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T31TCJ_B01.jp2')/$value"
>>>r=requests.get(url,auth=('hagolle', 'very_secret'))
>>>r.content '<?xml version=\'1.0\' encoding=\'UTF-8\'?><error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><code/><message xml:lang="en">An exception occured while creating the stream for node S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T31TCJ_B01.jp2</message></error>'

7

Olivier

Works for me. I use

r=requests.get(url, auth = (‘user’, ‘password’), verify=False)

because scihub got its SSL certificates all screwed up.

I get the JP2 file, but it’s only 54020 bytes, because there is only a small upper left corner with data (> 90% no data).

Guido

1 Like
8

Well, I have understood, there is an issue with my account, I created a new one and it works this time.
I will ask EOsupport again.
Thanks Guido !

9

Hello to all,
I have the same problem. I’d like to download a single granule folder of a L1C SE2 data but when I type on the address bar of my browser

https://scihub.copernicus.eu/dhus/odata/v1/Products(‘81acec7c-3020-408b-8125-9193b3975b28’)/Nodes(‘S2A_OPER_PRD_MSIL1C_PDMC_20160128T032144_R094_V20160125T111611_20160125T111611.SAFE’)/Nodes(‘GRANULE’)/Nodes(‘S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T30SWF_N02.01’)/$value

I obtain the answer:

An exception occured while creating the stream for node S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T30SWF_N02.01

I’ve already tried to create another account and I’ve tried to use the requests lib in python but nothing changed. I obtain the same answer to my download request.

Does anyone have any suggestion?
10

You’re looking at a directory node, so you should use Nodes() instead of value. Works in a browser.

11

I am sorry,
I did not understand.
you mean that the correct syntax should be:

https://scihub.copernicus.eu/dhus/odata/v1/Products(‘81acec7c-3020-408b-8125-9193b3975b28’)/Nodes(‘S2A_OPER_PRD_MSIL1C_PDMC_20160128T032144_R094_V20160125T111611_20160125T111611.SAFE’)/Nodes(‘GRANULE’)/Nodes(‘S2A_OPER_MSI_L1C_TL_SGS__20160125T164509_A003094_T30SWF_N02.01’)/Nodes()

12

Yes, correct. This will give you the XML that describes the node content (several directories and the XML file for the granule). You need to expand to the actual files you want to download.

13

Thanks for the answer. So in order to download a single granule should I have to download all the single files by using the folder hierarchy?

14

If you need all files then yes. But if you only need the JP2 files, you can go directly to Nodes(‘IMG_DATA’)/Nodes() and select the ones you need (e.g. maybe only the 10 m bands). Alternatively, you can pick up the cloud mask from QI_DATA, check whether your ROI is cloud free and then decide to yes/no download the bands. All depends on how you want to deal with the data and reduce e-waste :grin:

1 Like
15

Hello Everyone,

I am using the GitHub - olivierhagolle/Sentinel-download: Automated download of Sentinel-2 L1C data from ESA (through wget) :http://olivierhagolle.github.io/Sentinel-download script. And by the way thank you Olivier Hagolle for the great tool.

I have my command like this:

python C:\Python3\Scripts\Sentinel_download.py --lat 12.44966 --lon -87.05445 -a C:\Python3\Scripts\apihub2.txt -d 2016-01-16 -f 2016-01-31 -n -s S2
which runs well until I get an error message related with autentication. I get:

wget --no-check-certificate --user=“guest891” --password=“mypasswordAx” --output-document=query_results.xml “https://scihub.copernicus.eu/apihub/search?q=footprint:“Intersects(12.449660,-87.054450)” filename:S2* producttype:S2MSI1C &rows=100”
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\wget\GnuWin32/etc/wgetrc
–2017-08-04 23:12:55-- https://scihub.copernicus.eu/apihub/search?q=footprint:Intersects(12.449660,-87.054450)%20filename:S2*%20producttype:S2MSI1C%20&rows=100
Resolving scihub.copernicus.eu… 131.176.236.28
Connecting to scihub.copernicus.eu|131.176.236.28|:443… connected.
WARNING: cannot verify scihub.copernicus.eu’s certificate, issued by `/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2’:
Self-signed certificate encountered.
HTTP request sent, awaiting response… 401 Unauthorized
Reusing existing connection to scihub.copernicus.eu:443.
HTTP request sent, awaiting response… 401 Unauthorized
Authorization failed.
Traceback (most recent call last):
File “C:\Python3\Scripts\Sentinel_download.py”, line 280, in
xml=minidom.parse(“query_results.xml”)
File “C:\Python27\lib\xml\dom\minidom.py”, line 1918, in parse
return expatbuilder.parse(file)
File “C:\Python27\lib\xml\dom\expatbuilder.py”, line 924, in parse
result = builder.parseFile(fp)
File “C:\Python27\lib\xml\dom\expatbuilder.py”, line 211, in parseFile
parser.Parse(“”, True)
xml.parsers.expat.ExpatError: no element found: line 1, column 0

what is wrong with this? thank you for your help