Sen2three doesn't import all products in folder

s2tbx sen2three
#1

Hi, I have a set of 37 L2A products from April 2017 from different tiles in one folder. For most of the tiles, there are at least two products. The products were processed with sen2cor version 2.4.0 in the previous step with a target resolution of ten meters.
The products in the folder are: S2A_MSIL2A_20170401T100021_N0204_R122_T32TQS_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T32TQT_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33TUM_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33TUN_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33TVM_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33TVN_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33TWM_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33TWN_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33TXN_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33UUP_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33UVP_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33UVQ_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33UWP_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33UWQ_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33UXP_20170401T100022.SAFE S2A_MSIL2A_20170401T100021_N0204_R122_T33UXQ_20170401T100022.SAFE S2A_MSIL2A_20170407T102021_N0204_R065_T32TNS_20170407T102343.SAFE S2A_MSIL2A_20170407T102021_N0204_R065_T32TNT_20170407T102343.SAFE S2A_MSIL2A_20170407T102021_N0204_R065_T32TPS_20170407T102343.SAFE S2A_MSIL2A_20170407T102021_N0204_R065_T32TPT_20170407T102343.SAFE S2A_MSIL2A_20170407T102021_N0204_R065_T32TQT_20170407T102343.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T32TNS_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T32TNT_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T32TPS_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T32TPT_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T32TQS_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T32TQT_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T32UQU_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33TUM_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33TUN_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33TVM_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33TVN_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33UUP_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33UVP_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33UVQ_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33UWP_20170424T101120.SAFE S2A_MSIL2A_20170424T101031_N0204_R022_T33UWQ_20170424T101120.SAFE

When I run: L3_Process --resolution=10 C:\L2_Apr_17
only 3 products from different tiles are imported and it says that all tiles are already processed.

The output is: Sentinel-2 Level 3 Processor (Sen2Three), 1.1.0, created: 2017.07.01 started with 10m resolution ... Procedure L3_Tables: start import, elapsed time[s]: 7.860 Progress[%]: 1.16 : Procedure L3_Tables: Level L3 band B02 imported, elapsed time[s]: 51.348 Progress[%]: 8.77 : Procedure L3_Tables: Level L3 band B03 imported, elapsed time[s]: 46.910 Progress[%]: 15.73 : Procedure L3_Tables: Level L3 band B04 imported, elapsed time[s]: 47.337 Progress[%]: 22.74 : Procedure L3_Tables: Level L3 band B08 imported, elapsed time[s]: 50.993 Progress[%]: 30.30 : Procedure L3_Tables: Level L3 band AOT imported, elapsed time[s]: 1.217 Progress[%]: 30.48 : Procedure L3_Tables: Level L3 band SCL imported, elapsed time[s]: 27.799 Progress[%]: 34.60 : Procedure L3_Tables: start import, elapsed time[s]: 1.236 Progress[%]: 34.78 : Procedure L3_Tables: Level L3 band B02 imported, elapsed time[s]: 14.547 Progress[%]: 36.94 : Procedure L3_Tables: Level L3 band B03 imported, elapsed time[s]: 14.269 Progress[%]: 39.05 : Procedure L3_Tables: Level L3 band B04 imported, elapsed time[s]: 14.376 Progress[%]: 41.18 : Procedure L3_Tables: Level L3 band B08 imported, elapsed time[s]: 14.450 Progress[%]: 43.32 : Procedure L3_Tables: Level L3 band AOT imported, elapsed time[s]: 1.184 Progress[%]: 43.50 : Procedure L3_Tables: Level L3 band SCL imported, elapsed time[s]: 23.032 Progress[%]: 46.91 : Procedure L3_Tables: start import, elapsed time[s]: 1.178 Progress[%]: 47.09 : Procedure L3_Tables: Level L3 band B02 imported, elapsed time[s]: 14.333 Progress[%]: 49.21 : Procedure L3_Tables: Level L3 band B03 imported, elapsed time[s]: 14.572 Progress[%]: 51.37 : Procedure L3_Tables: Level L3 band B04 imported, elapsed time[s]: 14.873 Progress[%]: 53.57 : Procedure L3_Tables: Level L3 band B08 imported, elapsed time[s]: 15.612 Progress[%]: 55.89 : Procedure L3_Tables: Level L3 band AOT imported, elapsed time[s]: 1.181 Progress[%]: 56.06 : Procedure L3_Tables: Level L3 band SCL imported, elapsed time[s]: 23.019 Progress[%]: 59.47 : All tiles already processed. Application terminated successfully.

The processed-file is:

20170401T100022_T33UXQ_10 20170407T102343_T32TQT_10 20170424T101120_T33UWQ_10

I tried varying all parameters in L3_GIPP.xml and also tried it with products from different months, including products from 2015, but always had the same problem. L3_GIPP.xml currently looks like this:

<?xml version="1.0" encoding="UTF-8"?><Level-03_Ground_Image_Processing_Parameter xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="L3_GIPP.xsd"> <Common_Section> <Log_Level>INFO</Log_Level> <!-- NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL --> <Display_Data>true</Display_Data> <Target_Directory>DEFAULT</Target_Directory> <PSD_Scheme PSD_Version="13" PSD_Reference="S2-PDGS-TAS-DI-PSD-V13.1_Schema"> <UP_Scheme_2A>S2_User_Product_Level-2A_Metadata.xsd</UP_Scheme_2A> <UP_Scheme_3>S2_User_Product_Level-3_Metadata.xsd</UP_Scheme_3> <Tile_Scheme_2A>S2_PDI_Level-2A_Tile_Metadata.xsd</Tile_Scheme_2A> <Tile_Scheme_3>S2_PDI_Level-3_Tile_Metadata.xsd</Tile_Scheme_3> <DS_Scheme_2A>S2_PDI_Level-2A_Datastrip_Metadata.xsd</DS_Scheme_2A> <DS_Scheme_3>S2_PDI_Level-3_Datastrip_Metadata.xsd</DS_Scheme_3> </PSD_Scheme> <PSD_Scheme PSD_Version="14" PSD_Reference="S2-PDGS-TAS-DI-PSD-V14.2_Schema"> <UP_Scheme_2A>S2_User_Product_Level-2A_Metadata.xsd</UP_Scheme_2A> <UP_Scheme_3>S2_User_Product_Level-3_Metadata.xsd</UP_Scheme_3> <Tile_Scheme_2A>S2_PDI_Level-2A_Tile_Metadata.xsd</Tile_Scheme_2A> <Tile_Scheme_3>S2_PDI_Level-3_Tile_Metadata.xsd</Tile_Scheme_3> <DS_Scheme_2A>S2_PDI_Level-2A_Datastrip_Metadata.xsd</DS_Scheme_2A> <DS_Scheme_3>S2_PDI_Level-3_Datastrip_Metadata.xsd</DS_Scheme_3> </PSD_Scheme> <GIPP_Scheme>L3_GIPP.xsd</GIPP_Scheme> </Common_Section> <L3_Synthesis> <Min_Time>2016-01-01T00:00:00Z</Min_Time> <Max_Time>2017-12-31T23:59:59Z</Max_Time> <Tile_Filter>*</Tile_Filter> <Algorithm>AVERAGE</Algorithm> <!-- MOST_RECENT, TEMP_HOMOGENEITY, RADIOMETRIC_QUALITY, AVERAGE --> <Radiometric_Preference>AEROSOL_OPTICAL_THICKNESS</Radiometric_Preference> <!-- AEROSOL_OPTICAL_THICKNESS, SOLAR_ZENITH_ANGLE --> <Cirrus_Removal>true</Cirrus_Removal> <Shadow_Removal>true</Shadow_Removal> <Snow_Removal>false</Snow_Removal> <Max_Cloud_Probability>100.0</Max_Cloud_Probability> <Max_Invalid_Pixels_Percentage>100.0</Max_Invalid_Pixels_Percentage> <Max_Aerosol_Optical_Thickness>100.0</Max_Aerosol_Optical_Thickness> <Max_Solar_Zenith_Angle>70.0</Max_Solar_Zenith_Angle> <Median_Filter>3</Median_Filter> </L3_Synthesis> <Classificators> <NO_DATA>0</NO_DATA> <SATURATED_DEFECTIVE>1</SATURATED_DEFECTIVE> <DARK_FEATURES>2</DARK_FEATURES> <CLOUD_SHADOWS>3</CLOUD_SHADOWS> <VEGETATION>4</VEGETATION> <NOT_VEGETATED>5</NOT_VEGETATED> <WATER>6</WATER> <UNCLASSIFIED>7</UNCLASSIFIED> <MEDIUM_PROBA_CLOUDS>8</MEDIUM_PROBA_CLOUDS> <HIGH_PROBA_CLOUDS>9</HIGH_PROBA_CLOUDS> <THIN_CIRRUS>10</THIN_CIRRUS> <SNOW_ICE>11</SNOW_ICE> <URBAN_AREAS>12</URBAN_AREAS> </Classificators> </Level-03_Ground_Image_Processing_Parameter>

I’m using sen2three version 1.1.0 on Windows Server 2012 R2 Datacenter, but had the same problem with Windows 10 before.
Any help would be highly appreciated!

#2

Hi gepo1,

can you please:

  1. remove the created folder S2A_MSIL03_…
  2. remove the file processed
  3. start again the processing
  4. attach the file L3_2017…_report.xml, which is written in the log directory of the senthree home folder

Each product which is not present at least 2x, will not produce any output.
can you thus perhaps reduce the folder to the products containing:

T32TNS_20170407T102343.SAFE
T32TNS_20170424T101120.SAFE
in the first run,

and then to:
T32TNT_20170407T102343.SAFE
T32TNT_20170424T101120.SAFE
T32TPS_20170407T102343.SAFE
T32TPS_20170424T101120.SAFE
T32TPT_20170407T102343.SAFE
T32TPT_20170424T101120.SAFE
T32TQT_20170407T102343.SAFE
T32TQT_20170424T101120.SAFE
T33UUP_20170401T100022.SAFE
T33UUP_20170424T101120.SAFE

to see what is produced then?
In order to track down your problem.

Thanks and cheers,
umwilm

#3

Hi umwilm, when I run it again with all 37 products mentioned:
L3_20171120T133628_report.xml (12.6 KB)

When I run it with just the two products as you suggested, it seems to work alright:
L3_20171120T140223_report.xml (15.7 KB)

When I run it with the ten products as you suggested further, it produces an output for two of the tiles, one for T33UUP and one for T32TQT:L3_20171120T144326_report.xml (22.2 KB)

But the output for T32TQT seems to be placed in the location of tile T33UUP, too.

Thanks and cheers!