Sen2cor: KeyError: 'gfs'

Optical Toolbox sen2cor
#1

Hi,

when running sen2cor on this dataset:

  • File: S2A_OPER_PRD_MSIL1C_PDMC_20160723T092005_R090_V20160723T041553_20160723T041553.zip
  • UUID: 0d55c40f-3eca-4e0f-8fb2-482a5fa0585d

we get this error:

Traceback (most recent call last):
  File "/root/anaconda2/bin/L2A_Process", line 9, in <module>
    load_entry_point('sen2cor==2.2.1', 'console_scripts', 'L2A_Process')()
  File "/root/anaconda2/lib/python2.7/site-packages/sen2cor-2.2.1-py2.7.egg/sen2cor/L2A_Process.py", line 261, in main
    mn.generate(config.L2A_UP_DIR, config.L2A_MANIFEST_SAFE)
  File "/root/anaconda2/lib/python2.7/site-packages/sen2cor-2.2.1-py2.7.egg/sen2cor/L2A_Manifest.py", line 275, in generate
    self.addDataObject(dataObjectSection, filepath, ids[filepath], l2productpath)
  File "/root/anaconda2/lib/python2.7/site-packages/sen2cor-2.2.1-py2.7.egg/sen2cor/L2A_Manifest.py", line 147, in addDataObject
    byteStream.attrib["mimeType"] = self.getMimeType(href)
  File "/root/anaconda2/lib/python2.7/site-packages/sen2cor-2.2.1-py2.7.egg/sen2cor/L2A_Manifest.py", line 122, in getMimeType
    return dicoMimeType[href.split(".")[-1]]
KeyError: 'gfs'

In the log file, we see this:

cat ~/sen2cor/log/S2A_USER_PRD_MSIL2A_PDMC_20160723T092005_R090_V20160723T041553_20160723T041553.SAFE_report.xml
...
<inspection execution="2016-07-26 16:19:14,357" level="INFO" process="31805" module="L2A_XmlParser" function="validate" line="116"/>
<message contentType="Text">validating metadata file S2A_USER_MTD_L2A_TL_SGS__20160723T075653_A005664_T49VCF.xml against scheme</message>
</check>
<check>
<inspection execution="2016-07-26 16:19:14,407" level="INFO" process="31805" module="L2A_XmlParser" function="validate" line="122"/>
<message contentType="Text">metadata file is valid</message>
</check>
<check>
<inspection execution="2016-07-26 16:19:14,407" level="INFO" process="31805" module="L2A_XmlParser" function="validate" line="122"/>
<message contentType="Text">metadata file is valid</message>
</check>
<check>
<inspection execution="2016-07-26 16:19:15,270" level="INFO" process="31764" module="L2A_XmlParser" function="validate" line="116"/>
<message contentType="Text">validating metadata file S2A_USER_MTD_SAFL2A_PDMC_20160723T092005_R090_V20160723T041553_20160723T041553.xml against scheme</message>
</check>
<check>
<inspection execution="2016-07-26 16:19:15,337" level="ERROR" process="31764" module="L2A_XmlParser" function="validate" line="126"/>
<message contentType="Text">Schema file: S2-PDGS-TAS-DI-PSD-V13.1_Schema/S2_User_Product_Level-2A_Metadata.xsd</message>
</check>
<check>
<inspection execution="2016-07-26 16:19:15,337" level="ERROR" process="31764" module="L2A_XmlParser" function="validate" line="127"/>
<message contentType="Text">Details: Element 'BAND_NAME': This element is not expected.</message>
</check>

and we see that “band 8” is repeated multiple times:

/mnt/workspace/scratch/0d55c40f-3eca-4e0f-8fb2-482a5fa0585d/S2A_USER_PRD_MSIL2A_PDMC_20160723T092005_R090_V20160723T041553_20160723T041553.SAFE/S2A_USER_MTD_SAFL2A_PDMC_20160723T092005_R090_V20160723T041553_20160723T041553.xml
<n1:Level-2A_User_Product xmlns:n1="https://psd-13.sentinel2.eo.esa.int/PSD/User_Product_Level-2A.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://psd-13.sentinel2.eo.esa.int/PSD/User_Product_Level-2A.xsd">
  <n1:General_Info>
    <L2A_Product_Info>
      <PRODUCT_START_TIME>2016-07-23T04:15:53.456Z</PRODUCT_START_TIME>
      <PRODUCT_STOP_TIME>2016-07-23T04:15:53.456Z</PRODUCT_STOP_TIME>
      <PRODUCT_URI>http://www.telespazio-vega.de</PRODUCT_URI>
      <PROCESSING_LEVEL>Level-2Ap</PROCESSING_LEVEL>
      <PRODUCT_TYPE>S2MSI2Ap</PRODUCT_TYPE>
      <PROCESSING_BASELINE>02.04</PROCESSING_BASELINE>
      <GENERATION_TIME>2016-07-26T13:42:07Z</GENERATION_TIME>
      <PREVIEW_IMAGE_URL>http://www.telespazio-vega.de</PREVIEW_IMAGE_URL>
      <PREVIEW_GEO_INFO>BrowseImageFootprint</PREVIEW_GEO_INFO>
      <Datatake datatakeIdentifier="GS2A_20160723T041552_005664_N02.04">
        <SPACECRAFT_NAME>Sentinel-2A</SPACECRAFT_NAME>
        <DATATAKE_TYPE>INS-NOBS</DATATAKE_TYPE>
        <DATATAKE_SENSING_START>2016-07-23T04:15:52.026Z</DATATAKE_SENSING_START>
        <SENSING_ORBIT_NUMBER>90</SENSING_ORBIT_NUMBER>
        <SENSING_ORBIT_DIRECTION>DESCENDING</SENSING_ORBIT_DIRECTION>
      </Datatake>
      <Query_Options>
        <Area_Of_Interest>
          <Bbox>
            <LOWER_CORNER>57.156 -180.0000</LOWER_CORNER>
            <UPPER_CORNER>58.956 180.0000</UPPER_CORNER>
          </Bbox>
        </Area_Of_Interest>
        <FULL_SWATH_DATATAKE>false</FULL_SWATH_DATATAKE>
        <PREVIEW_IMAGE>true</PREVIEW_IMAGE>
        <Band_List>
          <BAND_NAME>B1</BAND_NAME>
          <BAND_NAME>B2</BAND_NAME>
          <BAND_NAME>B3</BAND_NAME>
          <BAND_NAME>B4</BAND_NAME>
          <BAND_NAME>B5</BAND_NAME>
          <BAND_NAME>B6</BAND_NAME>
          <BAND_NAME>B7</BAND_NAME>
          <BAND_NAME>B8</BAND_NAME>
          <BAND_NAME>B8</BAND_NAME>
          <BAND_NAME>B8</BAND_NAME>
          <BAND_NAME>B8</BAND_NAME>
          <BAND_NAME>B8</BAND_NAME>
          <BAND_NAME>B8</BAND_NAME>
          <BAND_NAME>B9</BAND_NAME>
          <BAND_NAME>B10</BAND_NAME>
          <BAND_NAME>B11</BAND_NAME>
          <BAND_NAME>B12</BAND_NAME>
          <BAND_NAME>B8A</BAND_NAME>
        </Band_List>
        <METADATA_LEVEL>Expertise</METADATA_LEVEL>
        <Aux_List productLevel="Level-2Ap">
...

Any idea what causes that? Thanks.

SEN2COR L2A_Process doesnt start?!
#2

I have repeatedly been getting this error in last 2 days on sen2cor v2.2.1 on several datasets over Australia, e.g. S2A_OPER_PRD_MSIL1C_PDMC_20160722T022900_R073_V20160722T001217_20160722T001217.SAFE

I did not pick up on the error in the meta data. Does manually deleting all but one of the BAND_NAME B8 lines fix the problem?

SNAP5.0, Sen2Cor 2.3.0 Error: Element 'BAND_NAME': This element is not expected. Is this the bug?
#3

I get this error even when XML looks OK (without the repeated band 8)

#4

Working on scene: S2A_OPER_PRD_MSIL1C_PDMC_20160721T120203_R060_V20160721T021833_20160721T021833.SAFE

It appears the error is being thrown because the .gfs file type is not listed as a file type in this line of code taken from L2A_Manifest.py:

def getMimeType(self, href):
    """
    Returns mime type function of file extension
    """
    dicoMimeType = { 'xml': "text/xml",
                    'gml': "application/xml",
                    'html': "octet",
                    'xsl': "octet",
                    'jp2': "application/octet-stream" }
    return dicoMimeType[href.split(".")[-1]]

This file throws the error for me:
GRANULE/S2A_USER_MSI_L2A_TL_SGS__20160721T090523_A005634_T50JLL_N02.04/QI_DATA/S2A_OPER_MSK_TECQUA_SGS__20160721T090523_A005634_T50JLL_B01_MSIL1C.gfs

Sentinel2 atmospheric correction
#5

So… temporary hack is to go into the L2A GRANULES/QI_DATA and delete all files that end with .gfs then re-run L2A_Process. It will recognise all bands have already been processed and finish the job successfully in a few seconds.

2 Likes
How to process sentinel2 using sen2cor
Can sentinel2 be processed big datausing sen2cor?
#6

A more permanent hack for those with automated systems is add gfs to the dictionary, so I changed the above code to:

def getMimeType(self, href):
“”"
Returns mime type function of file extension
“”"
dicoMimeType = { ‘xml’: “text/xml”,
‘gml’: “application/xml”,
‘html’: “octet”,
‘xsl’: “octet”,
‘jp2’: “application/octet-stream”,
‘gfs’: “gfs”}
return dicoMimeType[href.split(".")[-1]]

2 Likes
#7

Thanks for looking into this. Will you propose it to submit it as pull request so that it goes into the next release?

#8

I could but I do not think it is a complete solution. The *.gfs files have been added and all my modification does is assign them with a dummy ‘MimeType’ in the L2A_manifest.py so that the code will not throw error.

Surely there is someone at ESA who can look into this further.

#9

I think that it is a problem with N02.04 products ( https://sentinel.esa.int/web/sentinel/missions/sentinel-2/news/-/article/new-processing-baseline-02-04-for-sentinel-2a-products ).
If it is any solution for this problem, please reply for my post.

#10

I’m getting the same error and my output is coming without the Band 8. Did anyone know how to solve it ?

#11

do you have QQ number?

#12

hi，did you solved this error？
My QQ is 569838064.

could you please help me？