Sen2cor failing with corrupted SRTM zip

neteler · November 30, 2016, 3:38pm

I got a gdalDEM() error in sen2cor 2.3.0, when processing S2A_OPER_PRD_MSIL1C_PDMC_20161111T050139_R087_V20161109T231802_20161109T231802.zip

Process L2A_ProcessTile-1:
Traceback (most recent call last):
  File "/home/mundialis/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 135, in run
    if self.process_10() == False:
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 178, in process_10
    if not self.process_20():
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 157, in process_20
    return self.process()
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 196, in process
    if(self.preprocess() == False):
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 289, in preprocess
    if(self.tables.importBandList() == False):
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_Tables.py", line 1084, in importBandList
    demfile = self.gdalDEM_srtm()
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_Tables.py", line 1501, in gdalDEM_srtm
    src_ds = gdal.Open(srtmf_dst, GA_Update)
RuntimeError: No such file or directory

Indeed, SRTM tiles are there:

[mundialis@fedora23 logs]$ ls -la ~/sen2cor/dem/
total 986416
drwxrwxr-x. 3 mundialis mundialis     4096 Nov 30 10:24 .
drwxrwxr-x. 5 mundialis mundialis     4096 Nov 28 22:00 ..
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 30 10:10 srtm_21_04.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 30 10:10 srtm_22_04.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 28 22:21 srtm_37_03.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 28 22:20 srtm_37_04.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 28 22:22 srtm_38_03.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 28 22:22 srtm_38_04.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 29 11:24 srtm_39_03.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 29 12:22 srtm_40_03.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 28 22:36 srtm_44_06.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 29 17:58 srtm_63_19.tif
drwxrwxr-x. 2 mundialis mundialis     4096 Nov 29 17:58 srtm_63_20_tmp
-rw-rw-r--. 1 mundialis mundialis      231 Nov 29 17:58 srtm_63_20.zip
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 29 17:58 srtm_64_19.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 29 17:59 srtm_64_20.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 29 18:16 srtm_65_19.tif
-rw-rw-r--. 1 mundialis mundialis 72145062 Nov 29 18:16 srtm_65_20.tif
[mundialis@fedora23 logs]$

The reason is likely the broken SRTM file (srtm_63_20.zip got silently corrupted due to an temporary internet outage). Ideally the sen2cor process would refetch the tile when unzipping fails.

neteler · November 30, 2016, 4:07pm

Well, it crashes even with all broken SRTM files cleaned up. So I don’t really know what it is looking for (“RuntimeError: No such file or directory” )…

neteler · November 30, 2016, 4:21pm

… and when disabling the DEM support, it crashes as well:

Sentinel-2 Level 2A Prototype Processor (Sen2Cor), 2.3.0, created: 2016.11.18 started ...
selected resolution is 10 m.
Progress[%]: 0.03 : PID-9256, 20 m resolution must be processed first, elapsed time[s]: 0.546
Progress[%]: 0.05 : PID-9256, L2A_ProcessTile: processing with resolution 20 m, elapsed time[s]: 0.500
Progress[%]: 0.05 : PID-9256, L2A_ProcessTile: start of pre processing, elapsed time[s]: 0.000
Progress[%]: 0.05 : PID-9256, L2A_Tables: start import, elapsed time[s]: 0.041
Progress[%]: 0.22 : PID-9256, L2A_Tables: band B01 imported, elapsed time[s]: 3.482
Progress[%]: 1.29 : PID-9256, L2A_Tables: band B02 imported, elapsed time[s]: 22.345
Progress[%]: 2.34 : PID-9256, L2A_Tables: band B03 imported, elapsed time[s]: 21.746
Progress[%]: 3.38 : PID-9256, L2A_Tables: band B04 imported, elapsed time[s]: 21.765
Progress[%]: 3.70 : PID-9256, L2A_Tables: band B05 imported, elapsed time[s]: 6.655
Progress[%]: 4.03 : PID-9256, L2A_Tables: band B06 imported, elapsed time[s]: 6.755
Progress[%]: 4.36 : PID-9256, L2A_Tables: band B07 imported, elapsed time[s]: 6.954
Progress[%]: 4.69 : PID-9256, L2A_Tables: band B8A imported, elapsed time[s]: 6.915
Progress[%]: 4.85 : PID-9256, L2A_Tables: band B09 imported, elapsed time[s]: 3.320
Progress[%]: 4.99 : PID-9256, L2A_Tables: band B10 imported, elapsed time[s]: 2.920
Progress[%]: 5.30 : PID-9256, L2A_Tables: band B11 imported, elapsed time[s]: 6.382
Progress[%]: 5.61 : PID-9256, L2A_Tables: band B12 imported, elapsed time[s]: 6.405
Progress[%]: 5.61 : PID-9256, L2A_ProcessTile: start of Scene Classification, elapsed time[s]: 0.004
Progress[%]: 5.64 : PID-9256, Pre process   , elapsed time[s]: 0.658
Progress[%]: 5.67 : PID-9256, L2A_SC init   , elapsed time[s]: 0.620
Progress[%]: 5.71 : PID-9256, L2A_CSND_1_1  , elapsed time[s]: 0.979
Progress[%]: 5.76 : PID-9256, L2A_CSND_1_2  , elapsed time[s]: 1.024

    Process L2A_ProcessTile-1:
Traceback (most recent call last):
  File "/home/mundialis/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 135, in run
    if self.process_10() == False:
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 178, in process_10
    if not self.process_20():
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 157, in process_20
    return self.process()
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_ProcessTile.py", line 205, in process
    if(sc.process() == False):
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_SceneClass.py", line 805, in process
    if(self.tables.sceneCouldHaveSnow() == True):
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/L2A_Tables.py", line 2422, in sceneCouldHaveSnow
    if(aoi.max() > 0):
  File "/home/mundialis/anaconda2/lib/python2.7/site-packages/numpy/core/_methods.py", line 26, in _amax
    return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity

NicWayand · December 22, 2016, 6:23pm

Bump. Also getting this error. Any resolution??