Topographic Correction Error: DEM not found

adwiputra · March 2, 2021, 8:19pm

Hi,

I encountered an error when trying to apply topographic correction to sentinel 1c dataset using sen2cor 2.8 on Windows 10 64 bit.
The DEM tiles have been correctly downloaded, but the process did not finish completely due to an error:

[Error 3] The system cannot find the path specified: 'E:\\pers_plan\\ubc\\research\\data\\spatial_data\\imageries\\sentinel-2\\studySite\\2015\\l1c_SAFE\\S2A_MSIL2A_20150810T031536_N9999_R118_T48PVA_20210302T200548.SAFE\\GRANULE\\L2A_T48PVA_A000687_20150810T033200\\tmpna3dhq\\srtm_L2A_T48PVA_A000687_20150810T033200_dem_int16.tif'
Traceback (most recent call last):
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\\Lib\site-packages\sen2cor\L2A_Process.py", line 495, in main
    if tile.process() == True:
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\Lib\site-packages\sen2cor\L2A_ProcessTileToolbox.py", line 131, in process
    if not self.process_resolution(10):
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\Lib\site-packages\sen2cor\L2A_ProcessTileToolbox.py", line 145, in process_resolution
    if not self.process_resolution(20):
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\Lib\site-packages\sen2cor\L2A_ProcessTileToolbox.py", line 148, in process_resolution
    return self._process()
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\Lib\site-packages\sen2cor\L2A_ProcessTileToolbox.py", line 162, in _process
    if self.preprocess() == False:
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\Lib\site-packages\sen2cor\L2A_ProcessTileToolbox.py", line 251, in preprocess
    if(self.tables.importBandList() == False):
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\Lib\site-packages\sen2cor\L2A_Tables.py", line 1001, in importBandList
    demfile = self.gdalDEM_srtm()
  File "C:\Users\mardongki\.snap\auxdata\Sen2Cor-02.08.00-win64\Lib\site-packages\sen2cor\L2A_Tables.py", line 1851, in gdalDEM_srtm
    os.remove(srtmf_dst_int16)  # check if needed (JL)
WindowsError: [Error 3] The system cannot find the path specified: 'E:\\pers_plan\\ubc\\research\\data\\spatial_data\\imageries\\sentinel-2\\studySite\\2015\\l1c_SAFE\\S2A_MSIL2A_20150810T031536_N9999_R118_T48PVA_20210302T200548.SAFE\\GRANULE\\L2A_T48PVA_A000687_20150810T033200\\tmpna3dhq\\srtm_L2A_T48PVA_A000687_20150810T033200_dem_int16.tif'
Progress[%]: 100.00 : Application terminated with at least one error.

When I navigate to the directory where the dem_int16.tif should be, the file actually exists. However, the icon has a red exclamation mark at the bottom left corner:

Please find the associated L2A_GIPP.xml provided here just in case it is relevant.

gnwiii · March 3, 2021, 1:41pm

The file is probably corrupt. Sometimes a DEM download gets interrupted by internet glitches. Often you can recognize this because the file is too small, but a more reliable test is to use a program like tiffinfo or gdalinfo. If the file was truncated, wget[2] has a “continue” option that will attempt to download only the remaining data, but this depends on the server configuration. Loops in “shell” scripts are ideal for this sort of cleanup (for a few years, internet at my work gave ca 10% corrupt downloads so I’ve done many such cleanups) on linux or with WSL2 or Cygwin shell program on Windows. It can be useful to create a auxiliary files for each image. A one-liner in
bash (of dash or zsh) is:

for img in *.tif ; do gdalinfo "$img" > "${img%.tif}.gdalinfo" 2>&1 ; done

Then use grep or awk to generate a list of corrupt files from the .gdalinfo results to download. This nicely illustrates one of the underlying principles behind unix – “programming is mostly text processing” – that has carried over to linux (and macOS).

adwiputra · March 4, 2021, 4:46pm

Hi, thanks for your response! The files were indeed corrupted.

I ended up running the process successfully after I moved the directory to an upper level. It seemed that the output directory for the temporary DEM file was too long to be handled by the process correctly.

gnwiii · March 4, 2021, 6:47pm

Glad you found the problem. Windows 10 does have an option to increase the maximum file path limit, but I’m not sure it works with all software. It would be nice if downloading tools could check paths to make sure they aren’t too long before initiating downloads.