Sen2cor processing parameters missing in final product?

[adriaan]

Dear all,

TL;DR
Not all configuration options of sen2cor appear to be traceable in the final level 2A product. As a consequence, it is impossible to know is some correction is or is not applied to the final product. The IODD lists the solution to this, copy the GIPP into the level 2A product. However, this appears not to be the case.

Full story

sen2cor offers four corrections that can be enabled/disabled in L2A_GIPP.xml, as well as some parameters for these algorithms. However, not all processing parameters appear to be covered by the MTD_MSIL2A.xml, MTD_DS.xml or other metedata files of the level 2A products. As a consequence it appears to be impossible to reverse engineer the corrections and parameters applied during processing.

The IODD, PDD and SUM documents are vague on this at best. However, section 2.1.2 of the IODD states in an incomplete sentence that the GIPP file is “and [sic] subsequently copied into the AUX_DATA subfolder of the corresponding granule for documentation purposes.” Unfortunately, I can not find such copy in my products. Furthermore, the suggested output is not listed in Table 4 of the IODD either.

As a consequence it is impossible to verify if similar corrections are applied to all images in a time series. Without it is impossible to conclude if a difference is due to a difference in reflectance or due to different TOA to BOA processing.

Therefore, I tried to compile a list of corrections and their presence in the output. I would welcome any additions/corrections. For each section I have added a conclusion on the traceability of the settings/properties. This is all based on the configuration parameters of sen2cor 2.8, as available on the ESA STEP website. Products processed with sen2cor 2.8 change the processing baseline 9999, to indicate a user product (with potentially different settings). However, sen2cor 2.5.5 copies the baseline from the initial product and no such indication is available. This necessitates a copy of the configuration in the final level 2A product.

Common

• Trivial are the settings Generate_DEM_Output, Generate_TCI_Output, Generate_DDV_Output and Downsample_20_to_60 that can be traced by the output generated.

Scene classification

• Median_Filter
  Default 0.
• The presence of the ESA CCI auxiliary data can be found in MTD_MSIL2A.xml as values for ESACCI_WaterBodies_Map, ESACCI_LandCover_Map, ESACCI_SnowCondition_Map_Dir.
• The auxillary GlobalSnowMap.tiff is provided by default and reported as SNOW_CLIMATOLOGY_MAP in MTD_MSIL2A.xml.

The setting for Median_Filter is not referenced in MTD_MSIL2A.xml or another output.

Conclusion: if the classification is present, classification was applied. The use of the auxiliary files can be derived by their presence in MTD_MSIL2A.xml, given the tile is within the latitude limits of SRTM. There is no indication if the median filter was applied.

LUT

• Aerosol_Type
  Default: RURAL.
• Mid_Latitude
  Default: SUMMER.
• Ozone_Content
  Default: 331.

Some reverse engineering is necessary. In MTD_MSIL2A.xml there the filename of the LUT used (LUT_FILENAME). One can compare this value between images to ensure the same LUT was used (see notes on AUTO in section 3.2.2.2.2 of SUM, page 17). Fortunately the IODD lists the filename specification that allows us to derive the original settings from this single LUT_FILENAME value. Not clear from the IODD (section 32.4, page 28) is that the first character indicates both the temperature profile and ozone content. Letters F-K indicate the increasing ozone contents for summer profiles, and T-Y are winter profiles.

Conclusion: settings can be reverse engineered from the MTD_MSIL2A.xml.

Aerosol Optical Thickness

• Visibility
  Default: 40.0.

The exact inner workings of this algorithm are unknown to me. The SUM states that this is an initialization parameter. The consequences of changing this setting (another output, or just slower/faster convergence) are unclear to me. However, I can not find it in the metadata of the level 2A product.

Conclusion: setting is not preserved, but may not be relevant?

Water vapor correction

• WV_Correction
  Default: 1. The comments in the GIPP file lists options 0-4, however, the IODD specifies only 0 and 1. Judging by the SUM, the algorithm might have been changed, as there is no indication of multiple algorithms.
• WV_Watermask
  Default: 100.
• Smooth_WV_Map
  Default: 100.

Hypothesis: if correction is applied, the water vapor map is available. Otherwise not.

Conclusion

Cirrus correction

• Cirrus_Correction
  Default: FALSE. When applied, cirrus correction is only applied to output at 20 and 60 meter resolution, not to 10 meter resolution!
• WV_Threshold_Cirrus
  Default: 0.25.

There is no indication if the correction was applied in MTD_MSIL2A.xml. The value for THIN_CIRRUS_PERCENTAGE and the classification SC_THIN_CIRRUS are from the classification algorithm instead.

Conclusion: there is no indication if the correction was applied.

Terrain correction (DEM)

• DEM_Terrain_Correction
  Boolean, default TRUE, but necessary parameters not supplied (effectively FALSE).
  Accompanying parameters:
• DEM_Directory
  Local directory where the DEM can be found or will be cached (irrelevant to processing).
• DEM_Reference
  Default is PlanetDEM (90m) for ESA products, none for user products. SRTM 4.1 (90m) is suggested for user products (http://data_public:GDdci@data.cgiar-csi.org/srtm/tiles/GeoTIFF/), but not enabled by default. Moreover, the IODD states (Table 2, page 15): “Currently only the CGIAR 90 m resolution DEMs are supported, …” Therefore, a local or high resolution DEM is only possible with a workaround. This claim is not repeated in Table 23 (page 35) where the same variable is discussed.
  The setting can be found in MTD_MSIL2A.xml as PRODUCTION_DEM_TYPE.
• Altitude
  Default: 0.100 km. Height to be used in the absence of a DEM, the setting can not be traced in MTD_MSIL2A.xml.

If a DEM is provided, but DEM_Terrain_Correction is set to FALSE, the DEM is “… only used for scene classification and AOT.” (IODD, 2.3.9.2/Table 23)

Conclusion: there is no indication if the correction was applied, only if a DEM was available. A correction with a constant height can not be traced.

Terrain correction (BRDF)

• BRDF_Correction
  Default: 0 (L2A_GIPP.xml), however the IODD recommends 21 as default.
• BRDF_Lower_Bound
  Default: 0.22.

There is no reference to the BRDF configuration in MTD_MSIL2A.xml.

Conclusion: there is no indication if the correction was applied and what settings were used.

Surface reflectance

• Adj_Km
  Default: 1.000

There is no reference to the this parameter in MTD_MSIL2A.xml.

Conclusion: there is no indication of the value used during processing.

Other

• VIS_Update_Mode
  Default: 1 (‘variable visibility’)

Unknown setting (not mentioned in either IODD, PDD or SUM).

Conclusion

Most corrections appear to be untraceable. Worst case I will have to rerun all corrections to ensure a homogeneous data set. An unfavorable situation. However, this can easily be solved, by including a copy of the L2A_GIPP.xml file in the final product.

[adriaan]

This issue appears to be (partially) solved for user generated products in sen2cor 2.9. The L2A_GIPP used is now copied to the rep_info folder of the L2A product. Where AUTO settings are used, a little reverse engineering might be necessary, as discussed above.