Question about geometric relict in L2A converted images

Overview

I am trying to do my own conversion of L1C images to L2A due to ESA’s long-term archive system. I’m using Sen2Cor v2.9 on a Linux system to convert my images. I’m finding that there is an odd geometrical relict that appears to be coming from inclusion of the DEM, even though the DEM is fine. I’m not sure how to fix or mitigate it. Does anyone have any suggestions?

I have confirmed this behavior is solely due to the inclusion of the DEM, not the added inclusion of ESACCI-LC.

What makes this doubly confusing is this behavior below is illustrated for 2019-03-04 for the specified tile, and it can also be seen with 2019-03-29 (S2B) and 2020-01-21 (S2A). Surprisingly, this was not seen for 2019-09-23 (S2A). I have no idea why.

Method

Using the parameters set as noted below, I ran Sen2Cor on an L1C image with the SRTM DEM and ESACCI-LC add-on to create the L2A image, then I differenced that image with the L2A file directly from Copernicus. When I look at the differenced image, I saw odd geometry so I decided to look at the image bands on their own. Shown here are the 10m bands for My L2A conversion (the first panel shows my study points that I’m extracting values for).

Here are the same bands for the downloaded Copernicus L2A.

Note that though I’m only showing the 10m bands here, the behavior is consistent across the 20m and 60m bands as well (and also for AOT).

I double-checked the DEM file itself, which appears to be fine (this is 6_5x5_TIFs/srtm_56_08.tif). I can not upload extra media to this post because I am a new user.

Parameters

Data

  • I downloaded the L1C imagery for T47RLH for 2019-03-04 from Copernicus via Google Earth Engine here.
  • I also downloaded the L2A imagery for the same tile and date in order to compare results (see here for the download).

External add-ons

  • DEM is the SRTM one downloaded as per the Sen2Cor v2.9 user guide, page 21 (I would link the user guide here but as a new user I am unable to put >2 links).
    • \path\Sen2Cor-02.09.00-Linux64
  • I also downloaded the ESACCI-LC files and added to the correct folder as per the user guide, page 32.
    • \path\Sen2Cor-02.09.00-Linux64\lib\python2.7\site-packages\sen2cor\aux_data

GIP parameters

  • In the L2A_GIPP.xml file, the following are set:
    • DEM_Directory and DEM_Reference are both set to my DEM folder
    • Force_Exit_On_DEM_Error is set to TRUE
  • In the L2A_PB_GIPP.xml, the following is set:
    • Baseline_Version is set to 02.11 in order to match the same parameter in the L2A_PB_GIPP.xml file from the Copernicus L2A image.

Command line

This is the direct command I used to convert the L1C image to the L2A image I used to create the pictures shown above, following the convention of the user guide page 35.

  • /path/Sen2Cor-02.09.00-Linux64/bin/L2A_Process
  • --output_dir /path/s2Data/L2A_DEM_ESACCI/47RLH
  • --GIP_L2A /path/Sen2Cor-02.09.00-Linux64/L2A_GIPP.xml (this is moved from default)
  • --GIP_L2A_PB /path/Sen2Cor-02.09.00-Linux64/lib/python2.7/site-packages/sen2cor/cfg/L2A_PB_GIPP.xml
  • --debug
  • /path/s2Data/L1C/47RLH/S2A_MSIL1C_20190304T035641_N0207_R004_T47RLH_20190304T064548.SAFE
1 Like

For the official products a different DEM is used.


This might be the reason why you see the difference.
The DEM used is Planet DEM. But it is not freely available.

But actually I don’t know why the different DEM could lead could lead to the strong horizontal edge.

P.S. Thanks for the good documentation of you case. :+1:t3:

1 Like

Thanks @marpet for your response and linking to the paper. I figured there would be some differences due to the DEM, but yes, the horizontal edge is confusing me. Overall I wanted to make sure I wasn’t making some kind of processing mistake on my end.

Hi @marpet I think I’ve found the problem.

My labmates suggested that potentially the SRTM DEM I’m using wasn’t matching up with the S2 tile. So I tried the following: (R code below)

  • bring the SRTM DEM file into R (in the SRTM file structure, this is 6_5x5_TIFs/srtm_56_08.tif)
  • reproject the raster to UTM (match the projection of the S2 tile)
  • overlay the tile with the reprojected SRTM

The images below show me 2 things. First, Sen2Cor v2.9 is correctly cropping the SRTM DEM tile to the extent of the original Sentinel2 tile. Second, because the DEMs do not match up, that’s where the hard horizontal edge is coming from. This tells me that Sen2Cor is pulling in the SRTM tile from above (srtm_56_07) to stitch together, but the DEMs are not equal in value / do not overlap well.

What are your thoughts? How do you propose moving forward?

R code:

library(raster)

#read in L2A image Band 2 (for example) and the SRTM dem
L2A <- stack("L2A_filePath/47RLH/S2A_MSIL2A_20190304T035641_N0211_R004_T47RLH_20190304T074200.SAFE/GRANULE/L2A_T47RLH_A019306_20190304T040110/IMG_DATA/R10m/T47RLH_20190304T035641_B02_10m.jp2")
dem <- stack(file.path(filePath, "Sen2Cor-02.09.00-Linux64/6_5x5_TIFs/srtm_56_08.tif"))
dem <- projectRaster(dem, crs=crs(L2A))

> extent(dem)
class      : Extent 
xmin       : 80843.49 
xmax       : 604984.7 
ymin       : 2210975 
ymax       : 2771328 

> extent(L2A)
class      : Extent 
xmin       : 3e+05 
xmax       : 409800 
ymin       : 2690220 
ymax       : 2800020

SRTM DEM tile plotted over L2A tile extent

plot(L2A)
plot(dem, add=TRUE)

L2A tile extent plotted over SRTM DEM tile extent

plot(dem)
plot(L2A, add=TRUE)

I looked at the data.

Upper right is the result when selecting ‘Add Elevation Band’ in SNAP.
The two lower images are the original SRTM tiles reprojected to UTM.
When I do a mosaic of the two tiles I get also a smooth result:

From where have you taken the SRTM data?
A suggestion:
Use also ‘Add Elevation Band’ in SNAP for you products.
When you open the image the data will be downloaded to
<USER_HOME>/.snap/auxdata/dem/SRTM 3Sec
Extract the tif files from the zips in this folder. Maybe they do work better.

I downloaded the SRTM data from this site (SRTMv4.1.zip) as per the user guide and put the two constituent folders in my Sen2Cor folder.

Each SRTM tile still remains in a .zip folder; it was my impression that Sen2Cor unzips these and uses the .tif files as needed, but perhaps I’m wrong and that’s where the issue is occurring.

Thank you for your suggestion for SNAP, however I am using Sen2Cor standalone on a Linux system (via terminal) and there is no option for ‘Add Elevation Band’. Thoughts?

Maybe you can use another system?

Or you download them directly here:
https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_56_07.zip
https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_56_08.zip

But it can also be that sen2cor handles the data wrong.

Unfortunately the linux system is all I have at the moment.

Other test

Thank you for those ESA download files. I downloaded those and tried running Sen2Cor with them, and I also changed the parameter <Generate_DEM_Output>TRUE</Generate_DEM_Output> so I could inspect the created DEM.

The horizontal edge was still evident in the output files. So I checked the output DEM and this is the result

  • output DEM path is l2aFolder/47RLH/S2A_MSIL2A_20190304T035641_N0211_R004_T47RLH_20210520T125247.SAFE/GRANULE/L2A_T47RLH_A019306_20190304T040110/AUX_DATA/T47RLH_20190304T035641_DEM_10m.jp2
    image

I then checked my DEM file structure, and only 1 of the tif files was extracted, namely, the bottom part of this image.
image

Overall this indicates to me that Sen2Cor v2.9 on Linux terminal is not correctly extracting and fusing the necessary DEMs for the corresponding Sentinel 2 tile. How do you suggest I move forward? Is there someone at ESA I should notify?

Other notes

  1. Sen2Cor failed with creating the 20m and 60m DEMs. The only metadata parameter I changed from my previous times running the software was changing FALSE to TRUE in
    • <Generate_DEM_Output>TRUE</Generate_DEM_Output>
    • Despite this, I received the following error message:
Syntax error in metadata, see report file for details.
Schema file: S2-PDGS-TAS-DI-PSD-V14.6_Schema/S2_User_Product_Level-2A_Metadata.xsd
Details: Element 'IMAGE_FILE': [facet 'pattern'] The value 'GRANULE/L2A_T47RLH_A019306_20190304T040110/AUX_DATA/T47RLH_20190304T035641_DEM_20m' is not accepted by the pattern 'GRANULE/L2A_T[\w{Lu}_]{5}_A\d{6}_\d{8}T\d{6}/AUX_DATA/L2A_T[\w{Lu}_]{5}_\d{8}T\d{6}_[A-Z]{3}_(1|2|6)0m'.
  1. I would also like to note, based on my previous comment yesterday, that I tried unzipping the srtm tile folders prior to running Sen2Cor but then Sen2Cor failed with error message:
[Errno 2] No such file or directory: '/path/to/DEMs/srtm_56_08.zip'
Zipfile download error for DEM: srtm_56_08.zip
Application will terminate

I don’t know who should be informed. I think Uwe Müller-Wilm (@umwilm) is a good contact point. Also Fabrizio Ramoino (@Fabrizio_Ramoino) is often well informed.
@oana_hogoiu do you now who should be contacted?

By the way. I have to correct myself.

PlanetDEM is not used anymore for the provided S2A data. Now the Copernicus DEM@90m is used.

These are the related tiles:

More here: Copernicus Digital Elevation Model (DEM) - Registry of Open Data on AWS

Thank you for the correction about the DEM and providing the tiles. Unfortunately, it appears Sen2Cor v2.9 only accepts the SRTM tiles. I tried the following

  • download the Copernicus tiles you provided and put them in a folder.
  • change the appropriate parameters in L2A_GIPP.xml file to reflect that folder
  • attempt running Sen2Cor

I received the following error message:

Trying to retrieve DEM from URL /folderPath/Sen2Cor-02.09.00-Linux64/CopernicusDEM/ this may take some time ...
[Errno 2] No such file or directory: '/folderPath/Sen2Cor-02.09.00-Linux64/CopernicusDEM/srtm_56_08.zip'
Zipfile download error for DEM: srtm_56_08.zip
Application will terminate
import of band list failed
Module L2A_T47RLH_A019306_20190304T040110 failed
Progress[%]: 100.00 : Application terminated with at least one error.

Context at this stage

  1. Because of how Sen2Cor is coded internally, the only way I can use a DEM is by having the SRTM tiles.
  2. But, Sen2Cor (via command line) does not correctly merge the SRTM DEM tiles when it needs to.
  3. It is worth noting that in the Sen2Cor v2.9 user guide (page 21), it states that Sen2Cor can only use either the SRTM DEM or the PlanetDEM.

@umwilm, @Fabrizio_Ramoino, @oana_hogoiu may I please get your opinions/thoughts on this? It appears the only workaround I currently have is to merge the DEM tiles myself and create a faux-zip folder for each tile, as that is what Sen2Cor is looking for.

I tried manually merging the DEM tiles for my full study area in Myanmar (using gdal_merge, e.g. see here), then duplicated that full .tif file into 9 separate zip folders using the same file structure that SRTM uses. When I ran Sen2Cor again, this solved the strong horizontal edge.

This image is the difference of my L2A raster (merged DEM + ESACCI) and the directly-downloaded Copernicus L2A.

However, there are still outstanding issues

  • this still does not solve the fact that Sen2Cor - despite the newest user guide saying it can accept SRTM DEM or Planet DEM - internally looks for SRTM file structure only.
  • also, Sen2Cor (in its standalone, linux version) is clearly not merging constituent DEM files correctly. This workaround was made as a temporary fix, but otherwise this must be an internal issue.
  • There remain the issues regarding the L2A_GIPP file I have mentioned at this STEP forum thread.
  • is there a particular reason why the exact ESA parameters for running Sen2Cor are not made available for the public? There is good documentation for SNAP but for any major batch processing I must be using Sen2Cor on a linux system for the high-powered computer.

Hi @ABraun and @Jan, I was wondering if either of you would have insight into this issue? I was looking up this thread about using a recent sen2cor and saw you both were knowledgeable about the process.

Thanks!

Hello Ian @mcgregorian1
I shall raise your concerns to the L2A expertise.

I note that the SUM for the latest version of Sen2Cor (2.9.0) at http://step.esa.int/thirdparties/sen2cor/2.9.0/docs/S2-PDGS-MPC-L2A-SUM-V2.9.0.pdf says that “The Planet DEM is not downloadable for free and must be purchased.” So - to me - it makes sense that it will look for SRTM data by default.

I also note that the CGIAR website has been shuffled around, and the SRTM Tiles are now accessible via: https://srtm.csi.cgiar.org/srtmdata/

Cheers

Jan

S2 MPC Operations Manager

Hi @Jan thank you for your response.

Yes, I saw that as well regarding the Planet DEM. However, @marpet mentioned in this thread that S2A data is provided from the Copernicus DEM 90m. Possibly sen2cor could instead look for a base DEM folder, then based on an argument called DEM_Type the algorithm could look for either the SRTM file structure or the Copernicus file structure?

Thank you for the SRTM tile source. I was also using this site to help identify the tiles as it has an underlying map on it.

1 Like