Sen2cor 2.4.0 fails on a NON VEGETATED product

Hi,
Just to let you know sen2cor 2.4.0. with the standalone installer, on a centOS 7 computer failed when processing a product on a desertic site.

Product reference : S2A_OPER_PRD_MSIL1C_PDMC_20160416T225749_R121_V20160416T082220_20160416T082220.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20160416T122809_A004265_T36RXV_N02.01

Several other products fail in the time series over that site, but not all of them, for instance, not this one : S2A_OPER_PRD_MSIL1C_PDMC_20160526T164051_R121_V20160526T082013_20160526T082013.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20160526T120458_A004837_T36RXV_N02.02

Command line :
L2A_Process --resolution 10 S2A_OPER_PRD_MSIL1C_PDMC_20160416T225749_R121_V20160416T082220_20160416T082220.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_SGS__20160416T122809_A004265_T36RXV_N02.01

Error message
Sentinel-2 Level 2A Processor (Sen2Cor), 2.4.0, created: 2017.06.05 started …
selected resolution is 10 m.
Process L2A_ProcessTile-1:
Traceback (most recent call last):
File “/mnt/data/home/hagolleo/PROG/SEN2COR/Sen2Cor-2.4.0-Linux64/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/mnt/data/home/hagolleo/PROG/SEN2COR/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/sen2cor/L2A_ProcessTile.py”, line 135, in run
if self.process_10() == False:
File “/mnt/data/home/hagolleo/PROG/SEN2COR/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/sen2cor/L2A_ProcessTile.py”, line 168, in process_10
self.config.readPreferences()
File “/mnt/data/home/hagolleo/PROG/SEN2COR/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Config.py”, line 4025, in readPreferences
par = node.NOT_VEGETATED
File “src/lxml/lxml.objectify.pyx”, line 229, in lxml.objectify.ObjectifiedElement.getattr (src/lxml/lxml.objectify.c:3837)
File “src/lxml/lxml.objectify.pyx”, line 450, in lxml.objectify._lookupChildOrRaise (src/lxml/lxml.objectify.c:6541)
AttributeError: no such child: NOT_VEGETATED

It’s true it is not vegetated…

Hi Olivier, I’ll create a ticket and let you know. Cheers,
Uwe

This is a simple read of a configuration parameter, which obviously fails. Can you attach the following file from your installation?
Sen2Cor-2.4.0-Linux64\lib\python2.7\site-packages\sen2cor\cfg\L2A_SC_GIPP.xml
Thanks, Uwe

I used the standard one without changing anything.
Here is the file content.

 <?xml version="1.0"  encoding="UTF-8"?>
<Level-2A_CAL_SC_Ground_Image_Processing_Parameter xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="L2A_CAL_SC_GIPP.xsd">
  <Scene_Classification>
    <References>
      <Snow_Map>GlobalSnowMap.tiff</Snow_Map>
    </References>
    <Classificators>
      <NO_DATA>0</NO_DATA>
      <SATURATED_DEFECTIVE>1</SATURATED_DEFECTIVE>
      <DARK_FEATURES>2</DARK_FEATURES>
      <CLOUD_SHADOWS>3</CLOUD_SHADOWS>
      <VEGETATION>4</VEGETATION>
      <NOT_VEGETATED>5</NOT_VEGETATED>
      <WATER>6</WATER>
      <UNCLASSIFIED>7</UNCLASSIFIED>
      <MEDIUM_PROBA_CLOUDS>8</MEDIUM_PROBA_CLOUDS>
      <HIGH_PROBA_CLOUDS>9</HIGH_PROBA_CLOUDS>
      <THIN_CIRRUS>10</THIN_CIRRUS>
      <SNOW_ICE>11</SNOW_ICE>
    </Classificators>
    <Thresholds>
      <T1_B02>0.18</T1_B02>
      <T2_B02>0.22</T2_B02>
      <T1_B04>0.06</T1_B04>
      <!-- modif JL 20151223 (was 0.08)-->
      <T2_B04>0.25</T2_B04>
      <T1_B8A>0.15</T1_B8A>
      <T2_B8A>0.35</T2_B8A>
      <T1_B10>0.012</T1_B10>
      <T2_B10>0.035</T2_B10>
      <T1_B12>0.25</T1_B12>
      <T2_B12>0.12</T2_B12>
      <T_B02_B12>0.018</T_B02_B12>
      <T_CLOUD_LP>0.0</T_CLOUD_LP>
      <T_CLOUD_MP>0.20</T_CLOUD_MP>
      <!-- modif JL 20151218 (was 0.35)-->
      <T_CLOUD_HP>0.80</T_CLOUD_HP>
      <!-- modif JL 20151218 (was 0.65)-->
      <T1_NDSI_CLD>-0.24</T1_NDSI_CLD>
      <T2_NDSI_CLD>-0.16</T2_NDSI_CLD>
      <T1_NDSI_SNW>0.35</T1_NDSI_SNW>
      <!-- modif JL 20151218 (was 0.20)-->
      <T2_NDSI_SNW>0.50</T2_NDSI_SNW>
      <!-- modif JL 20151218 (was 0.42)-->
      <T1_NDVI>0.36</T1_NDVI>
      <T2_NDVI>0.42</T2_NDVI>
      <!-- modif JL 20151218 (was 0.47 then 0.40)-->
      <T1_SNOW>0.12</T1_SNOW>
      <T2_SNOW>0.25</T2_SNOW>
      <T1_R_B02_B04>0.85</T1_R_B02_B04>
      <T2_R_B02_B04>0.95</T2_R_B02_B04>
      <T1_R_B8A_B03>1.50</T1_R_B8A_B03>
      <T2_R_B8A_B03>2.50</T2_R_B8A_B03>
      <T1_R_B8A_B11>0.90</T1_R_B8A_B11>
      <T2_R_B8A_B11>1.10</T2_R_B8A_B11>
      <T11_B02>-0.40</T11_B02>
      <T12_B02>0.46</T12_B02>
      <T11_R_B02_B11>0.70</T11_R_B02_B11>
      <!-- modif JL 20151218 (was 0.8 then 0.55)-->
      <T12_R_B02_B11>1.0</T12_R_B02_B11>
      <!-- modif JL 20151218 (was 1.5 then 0.8)-->
      <T21_R_B02_B11>2.00</T21_R_B02_B11>
      <T22_R_B02_B11>4.00</T22_R_B02_B11>
      <T21_B12>0.1</T21_B12>
      <T22_B12>-0.09</T22_B12>
      <RV_B2>6.96000</RV_B2>
      <RV_B3>5.26667</RV_B3>
      <RV_B4>5.37708</RV_B4>
      <RV_B8>7.52000</RV_B8>
      <RV_B11>5.45000</RV_B11>
      <RV_B12>2.55000</RV_B12>
      <T_SDW>0.75</T_SDW>
    </Thresholds>
  </Scene_Classification>
</Level-2A_CAL_SC_Ground_Image_Processing_Parameter>

Hi Olivier,
processing up to 10 m with the tile you flagged as failed went through at my test system without the problem you mentioned. Next question: did you choose parallel processing on tile level for the product? Anf if, how many processes? Can you attach the L2A GIPP.xml ? Thanx and Cheers, Uwe

Thanks Uwe,
Thanks for the inquiry !
I used the standard L2A GIPP.xml provided with the installation package. It is attached here.

I do not fully need that product, so if I am the only one to see that issue, do not spend to much time on it.L2A_GIPP.xml (4.5 KB)

Cheers,
Olivier

The error is strange nevertheless, 1. You showed that the configuration is not corrupt, as the entry NOT_VEGETATED is still there. 2. it’s a simple read of a parameter, no idea why this can fail. If it occurrs more often I need to reinvestigate, but in fact I never observed this issue until now. So I will not continue on this for the moment as I cannot reproduce. OK?

just thinking where to pin this issue…

can you post your PATH, PYTHONPATH and LD_LIBRARY_PATH variable ?

@OHagolle, do you have local python site-packages in ~/.local/ or somewhere to see if there is another lxml package that is getting used. I am just guessing the issue. To really test it you can edit ./bin/L2A_Process and add -s flag to python
$OUT_DIR/bin/python2.7 -s $SEN2COR_BIN/L2A_Process.py “$@”

Thanks Rashad,
i added the -s , but it gave the same result (AttributeError: no such child: NOT_VEGETATED). I have things in my path, but nothing special. And what is strange is that sen2cor works for many products except for a few of them on that site, and even on that site, some products are processed.
But as I said, I have enough results. This bug can be set aside until someone else stumbles upon it.
Best regards,
Olivier

yes, OK ! Thanks Uwe

Same error in windows 10 64 bits, stand alone instalation

can you send config xml?. I can have a look.

Here you are, thank you
L2A_CAL_SC_GIPP.xml (2.7 KB)

thanks. I will look into it. Maybe there is an issue with installed lxml package.
@OHagolle, had same issue but on linux. So i thnk this is not related to *nix packages.

I had a same problem after I updated from sen2cor-2.3.0 to sen2cor-2.4.0.

It works for me when I applied new L2A_Bashrc to update environment variables.

source L2A_Bashrc

can you post output of your L2A_Process ?

Hi,

I had the same problem, but noticed that it happened when I tried to process the tiles that were already processed with an older version of Sen2Cor (embedded in anaconda2). So I deleted the corresponding L2A products and tried again, this time without problems. Maybe this is an isolated case, I’ll provide more info if error occurs again.

Platform is Linux, I switched to standalone installation.

Cheers,
Ognjen