Hello Sen2Like team
We encountered some problems with sen2like. I will try to be as brief as possible
sen2like version is: sen2like.py 3.1.2
We are launching sen2like like this:
#:python sen2like.py single-tile-mode 36NXF --conf "/data/config.ini" --start-date 2019-01-01 --end-date 2019-12-31 --wd "/data/workingdir"
The config.ini:
[Processing]
doStitching = True
doGeometryKLT = True
doToa = True
doAtmcor = True
doNbar = True
doSbaf = True
doFusion = True
doPackager = True
doPackagerL2H = True
doPackagerL2F = True
The input data:
- Products from 2019/01/01 to 2019/12/31
- L8 Level 1 Path 170 ; Rows 58,59,60,61
- S2 L1C for 36NXF tile
- Reference -> S2 L2A full product from 20191004 with cloud coverage < 5 % (maybe this info is helpful for you), found in /data/References/reference_map.json:
β36NXFβ: β/data/References/S2B_MSIL2A_20191004T074749_N0213_R135_T36NXF_20191004T114035.SAFE/GRANULE/L2A_T36NXF_A013460_20191004T080733/IMG_DATA/R10m/T36NXF_20191004T074749_B
04_10m.jp2β
Problems noticed:
- I saw a RuntimeWarning, division by 0 at the beginning of the processing. Here is the log:
[INFO ] 2021-02-18 15:16:55 - reader - Sentinel2MTL Class
[INFO ] 2021-02-18 15:16:55 - reader - Product: /data/PRODUCTS/Sentinel2/36NXF/S2A_MSIL1C_20190102T075321_N0207_R135_T36NXF_20190102T092351.SAFE
/usr/local/sen2like_312/sen2like/sen2like/sen2like/atmcor/get_s2_angles.py:55: RuntimeWarning: invalid value encountered in true_divide
N = M / CPT
[INFO ] 2021-02-18 15:17:07 - sentinel2 - SAT_AZ , SAT_ZENITH, SUN_AZ, SUN_ZENITH
[INFO ] 2021-02-18 15:17:07 - sentinel2 - UNIT = DEGREES (scale: x100) :
[INFO ] 2021-02-18 15:17:07 - sentinel2 - /data/workingdir/47220/S2A_MSIL1C_20190102T075321_N0207_R135_T36NXF_20190102T092351.SAFE/tie_points.tif
[INFO ] 2021-02-18 15:17:07 - sentinel2 - Generating nodata mask from band B01
[INFO ] 2021-02-18 15:17:08 - image_file - Written: /data/workingdir/47220/S2A_MSIL1C_20190102T075321_N0207_R135_T36NXF_20190102T092351.SAFE/nodata_pixel_mask_B01.\
tif
[INFO ] 2021-02-18 15:17:08 - sentinel2 - Generating validity mask from cloud mask
[INFO ] 2021-02-18 15:17:08 - sentinel2 - Written: /data/workingdir/47220/S2A_MSIL1C_20190102T075321_N0207_R135_T36NXF_20190102T092351.SAFE/valid_pixel_mask.tif
Looking inside the source code, inside the sen2like/atmcor/get_s2_angles.py file, I noticed that there is no protection against division by 0 or NaN for line 55 (matrix division)
N = M / CPT
At line 48 there is a test if value is not nan for A matrix. If for a [i][j] postion is not fulfilled, that CPT[i][j] will remain 0, thus a later division by 0. I put a debug message for CPT when the RuntimeWarning appears:
# (time python sen2like.py single-tile-mode 36NXF --conf "/data/config.ini" --start-date 2019-01-01 --end-date 2019-12-31 --wd "/data/workingdir" ) |& tee /data/logs/36NXF_debug.log
[INFO ] 2021-02-19 08:51:06 - reader - Sentinel2MTL Class
[INFO ] 2021-02-19 08:51:06 - reader - Product: /data/PRODUCTS/Sentinel2/36NXF/S2A_MSIL1C_20190102T075321_N0207_R135_T36NXF_20190102T092351.SAFE
[ERROR ] 2021-02-19 08:51:18 - get_s2_angles - FOUND 0 in CPT: [[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 2. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1.]
[1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 2. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 2. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2.]
[1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 2. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 1.]
[1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[2. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1.]
[2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 2. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]]
/usr/local/sen2like_312/sen2like/sen2like/sen2like/atmcor/get_s2_angles.py:64: RuntimeWarning: invalid value encountered in true_divide
N = M / CPT
As you can see, there is a value of 0 inside the CPT, thus the runtime warning. I donβt know if this will impact things later
Also, maybe you should have a look at
M = np.zeros([x_size, y_size], np.float)
# print('input M :' + str(M[2][6]))
CPT = np.zeros([x_size, y_size], np.float)
for k, u in list(a_dict.items()):
for i in range(0, x_size, 1):
for j in range(0, x_size, 1):
A = u["Values"]
if A[i][j] == A[i][j]: # test if value is not nan
M[i][j] = A[i][j] + M[i][j]
CPT[i][j] += 1
M and CPT are [x_size, y_size] matrices , but later the parsing for y axis is done after x_size. I understand that this matrices are in fact squares, but one may be misled when following the code:
for i in range(0, x_size, 1):
for j in range(0, x_size, 1):
- I saw the same runtime warning for sen2like/s2l_processes/S2L_Fusion.py:
S2L_Fusion.py:309: RuntimeWarning: invalid value encountered in true_divide
Line 309 is :
A = (array2 - array1) / (doy_2 - doy_1)
I didnβt go deeper, but I will if needed, just let me know.
- I saw all the time the changing of the reference image, and I canβt understand why. I saw in your user manual the recommended reference image should be B04 with 10m. Instead, following the log, this is changed to the 30m B04:
[INFO ] 2021-02-18 15:39:31 - S2L_GeometryKLT - Start
[INFO ] 2021-02-18 15:39:31 - S2L_GeometryKLT - MGRS Framing: Start...
[INFO ] 2021-02-18 15:39:32 - S2L_GeometryKLT - MGRS Framing: End
[INFO ] 2021-02-18 15:39:32 - S2L_GeometryKLT - Change reference image to:/data/References/S2B_MSIL2A_20191004T074749_N0213_R135_T36NXF_20191004T114035.SAFE/GRANULE/L2A\
_T36NXF_A013460_20191004T080733/IMG_DATA/R10m/T36NXF_20191004T074749_B04_30m.TIF
[INFO ] 2021-02-18 15:39:32 - S2L_GeometryKLT - End
- I can see an error I canβt understand it yet:
[ERROR ] 2021-02-18 15:39:00 - hls_product - Error: Product band B02 with res 30 not found in /data/HLS_ts_scenario_1_36XNF/36NXF/S2A_MSIL2F_20190112T075301_N9999_R1\
35_T36NXF_20190112T092302.SAFE
- For the output, there are a lot of Landsat 8 L2Fproducts with the corrupted rasters, full of 0 or NaNs. For example, gdalinfo with stats for one of these products:
gdalinfo -stats /data/HLS_ts_scenario_1_36XNF/36NXF/LS8_OLIL2F_20190114T075459_N9999_R170_T36NXF_20190131T145256.SAFE/GRANULE/L2F_T36NXF_A000000_20190131T145256_LS8_R170/IMG_DATA/L2F_T36NXF_20190114T075459_LS8_R170_B02_10m.TIF
Driver: GTiff/GeoTIFF
Files: /data/HLS_ts_scenario_1_36XNF/36NXF/LS8_OLIL2F_20190114T075459_N9999_R170_T36NXF_20190131T145256.SAFE/GRANULE/L2F_T36NXF_A000000_20190131T145256_LS8_R170/IMG_DATA/L2F_T36NXF_20190114T075459_LS8_R170_B02_10m.TIF
/data/HLS_ts_scenario_1_36XNF/36NXF/LS8_OLIL2F_20190114T075459_N9999_R170_T36NXF_20190131T145256.SAFE/GRANULE/L2F_T36NXF_A000000_20190131T145256_LS8_R170/IMG_DATA/L2F_T36NXF_20190114T075459_LS8_R170_B02_10m.TIF.aux.xml
Size is 10980, 10980
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 36N",
......
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 600000.000, 100020.000) ( 33d53'55.50"E, 0d54'17.28"N)
Lower Left ( 600000.000, -9780.000) ( 33d53'55.10"E, 0d 5'18.50"S)
Upper Right ( 709800.000, 100020.000) ( 34d53' 7.12"E, 0d54'15.91"N)
Lower Right ( 709800.000, -9780.000) ( 34d53' 6.29"E, 0d 5'18.36"S)
Center ( 654900.000, 45120.000) ( 34d23'31.00"E, 0d24'29.14"N)
Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Gray
Min=0.000 Max=0.000
Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
Overviews: 5490x5490
Offset: 0, Scale:0.0001
Metadata:
STATISTICS_MAXIMUM=0
STATISTICS_MEAN=0
STATISTICS_MINIMUM=0
STATISTICS_STDDEV=0
STATISTICS_VALID_PERCENT=100
As you can see, the STATISTICS_ are 0.
Here is the sen2like log for this output product:
[INFO ] 2021-02-18 15:38:52 - S2L_Fusion - Start
[WARNING ] 2021-02-18 15:38:54 - hls_product - Product mask not found at /data/HLS_ts_scenario_1_36XNF/36NXF/S2A_MSIL2F_20190112T075301_N9999_R135_T36NXF_20190112T0923\
02.SAFE/S2A_MSIL2F_20190112T075301_N9999_R135_T36NXF_20190112T092302.SAFE_MSK.TIF
[INFO ] 2021-02-18 15:38:54 - hls_product - Searching it with S2Like format
[INFO ] 2021-02-18 15:38:54 - hls_product - Product mask finally found
[INFO ] 2021-02-18 15:38:56 - S2L_Fusion - prediction
[ERROR ] 2021-02-18 15:39:00 - hls_product - Error: Product band B02 with res 30 not found in /data/HLS_ts_scenario_1_36XNF/36NXF/S2A_MSIL2F_20190112T075301_N9999_R1\
35_T36NXF_20190112T092302.SAFE
[ERROR ] 2021-02-18 15:39:00 - hls_product -
[INFO ] 2021-02-18 15:39:00 - S2L_Fusion - Resampling to 30m: Start...
[INFO ] 2021-02-18 15:39:04 - S2L_Fusion - Resampling to 30m: End
/usr/local/sen2like_312/sen2like/sen2like/sen2like/s2l_processes/S2L_Fusion.py:309: RuntimeWarning: invalid value encountered in true_divide
A = (array2 - array1) / (doy_2 - doy_1)
[INFO ] 2021-02-18 15:39:17 - S2L_Fusion - fusion
[INFO ] 2021-02-18 15:39:23 - S2L_Fusion - End
[INFO ] 2021-02-18 15:39:25 - image_file - Written: /data/HLS_ts_scenario_1_36XNF/36NXF/L2F_36NXF_20190114_LS8_R170/L2F_36NXF_20190114_LS8_R170_B02_10m.TIF
[INFO ] 2021-02-18 15:39:25 - image_file - Written: /data/HLS_ts_scenario_1_36XNF/36NXF/L2F_36NXF_20190114_LS8_R170/L2F_36NXF_20190114_LS8_R170_B02_30m.TIF
[INFO ] 2021-02-18 15:39:27 - image_file - Written: /data/HLS_ts_scenario_1_36XNF/36NXF/LS8_OLIL2F_20190114T075459_N9999_R170_T36NXF_20190131T145256.SAFE/GRANULE/L\
2F_T36NXF_A000000_20190131T145256_LS8_R170/IMG_DATA/L2F_T36NXF_20190114T075459_LS8_R170_B02_10m.TIF
I can see in the log there are some links to the problems I described before: 2 and 4
I know itβs a lot of info here, but maybe you can have some hints, thank you so much for all your help you can provide !