iCOR sentinel2 AOT issue

bbrauzzi · January 30, 2019, 10:25am

Hi,
I am using the icor.py script from command line on a Centos7 machine with python 2.7.5 and I installed the “ICOR UPDATE FOR SNAP6.0” patch.
When I run the script with landsat8 products everything works fine, but with sentinel2 products I am encountering some issues during the Aerosol Optical Thickness phase. The process finishes successfully only if I set the aot parameter to false.
I tried using both sentinel2’s products released before and after the 2017 naming convention change. Tried both S2A and S2B.
I tried to shorten the s2 product’s path as suggested in iCOR Landsat 8 error

This is the command I am running

/usr/bin/python /opt/vito/icor/src/icor.py \
--keep_intermediate false \
 --cloud_average_threshold 0.19  \
 --cloud_low_band B01  \
 --cloud_low_threshold 0.25  \
 --cirrus true  \
 --aot true  \
 --aot_window_size 100  \
 --simec false  \
 --watervapor false  \
 --bg_window 1 \
 --cirrus_threshold 0.01  \
 --aot_override 0.1  \
 --ozone_override 0.33  \
 --wv_override 2.0  \
 --water_band B08  \
 --water_threshold 0.05  \
 --data_type S2  \
 --output_file /tmp/icorTest/output/S2A_*_processed.tif  \ 
/tmp/data/S2A_MSIL1C_20170101T121632_N0204_R123_T24FWG_20170101T121634.SAFE/GRANULE/L1C_T24FWG_A007985_20170101T121634/IMG_DATA/T24FWG_20170101T121632_B08.jp2

and this is the error I am getting.

======================================================================
 62% Estimate the Aerosol Optical Thickness (AOT)
======================================================================
Running ac_runner /tmp/atcconfidXCFh.conf
Running ac_runner /tmp/atcconftNMb1e.conf


0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************

Failed!
Process output:
 +---------------------------------------------------------------------
 | :: load config file /tmp/atcconftNMb1e.conf
 | :: open image : /tmp/icor_sV3oZU_proc/S2A_OPER_MSI_L1C_TL_MTI__20170103T052120_A007985_T24FWG_N02_04/S2A_OPER_MSI_L1C_TL_MTI__20170103T052120_A007985_T24FWG_N02_04_AOT_60M.tif
 | :: image does not contain valid values
 +---------------------------------------------------------------------

Thank you in advance

Blasco

icor · January 30, 2019, 12:53pm

Hello,

what is the surface content of the images ? Only water, clouds … ? It appears that the automatic AOT retrieval fails due to the absence of land (vegetation and soil) inside the image. Also, when the images are too cloudy, the process might fail.

greetings,
Stefan

bbrauzzi · January 31, 2019, 9:05am

Thank you for your prompt reply.
The issue was indeed the image content which only had clouds.

Thank you very much!
greetings

Blasco

binbinjiang · February 15, 2022, 2:36pm

Dear Stefan,

how could i output only for resolution = 60m? could you give me a hand. thanks

best

binbin

binbinjiang · February 17, 2022, 11:11pm

i also have same error in ICOR 3 run sentinel2

best

binbin

icor installation folder /opt/vito/icor
running icor for Sentinel2 ...
reading config /opt/vito/icor/src/config/local_sentinel2_simec.ini
output path = /home/binbin/Append/oke2018_2020/icor/S2A_MSIL1C_20181008T160511_N0206_R054_T17RNK_20181008T205913_processed.tif
iCOR temp folder : /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc
workflow = simec
/home/binbin/Append/oke2018_2020/orgin2018/S2A_MSIL1C_20181008T160511_N0206_R054_T17RNK_20181008T205913.SAFE/MTD_MSIL1C.xml
processing Sentinel2 granule : S2A_OPER_MSI_L1C_TL_MTI__20181008T205913_A017211_T17RNK_N02.06
======================================================================
  4% Convert to scaled radiance
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfosbrey4x.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf1cib4zbe.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfrhxbsuqv.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfbsjz45n_.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf4yv3eb7s.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfyfak019i.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf4wwyvwmv.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfm34x9fmj.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfygbc6flp.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfcwykbqfp.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfkqgs3okg.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfdz2njjm2.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfyo8ua9pv.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfqlvbakw6.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfv_fm6m60.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf13p_dfxb.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfbl6uaz6m.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfa604j0nv.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf_8pd1how.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfqdoorsed.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfm9b9_l09.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf52bflzfi.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfnkv5kitx.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconffoxhhn4y.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfh_3nxxvv.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfz24rqkzz.conf
======================================================================
  9% Generate DEM
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf2_3uzs7z.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfsb3fvhbq.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfkpm4d6x0.conf
======================================================================
 13% Resize images to 60m
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfxuns5lir.conf
======================================================================
 17% Single to MultiBand Radiance - 60M ALL
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfti0ocykt.conf
======================================================================
 22% Single to MultiBand Radiance - 60M 
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf54fm2bf6.conf
======================================================================
 26% Single to MultiBand Radiance - 20M
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfkf0qlh93.conf
======================================================================
 30% Single to MultiBand Radiance - 10M
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf7nucg_rx.conf
======================================================================
 35% Single to MultiBand Reflectance - 60M ALL
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfacdjc_d4.conf
======================================================================
 39% Cloud detection  - 60M
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf6rv8972q.conf
======================================================================
 43% Water detection
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfw58y1x5g.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconf17_k6k_0.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfspzu20ap.conf
======================================================================
 48% Estimate the Aerosol Optical Thickness (AOT)
======================================================================
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconfla1jc8f7.conf
Running ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconffu58njrc.conf
Failed!
Process output:
 +---------------------------------------------------------------------
 | b':: load config file /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconffu58njrc.conf'
 | b':: open image : /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/S2A_OPER_MSI_L1C_TL_MTI__20181008T205913_A017211_T17RNK_N02_06/S2A_OPER_MSI_L1C_TL_MTI__20181008T205913_A017211_T17RNK_N02_06_AOT_60M.tif'
 | b''
 | b''
 | b'0%   10   20   30   40   50   60   70   80   90   100%'
 | b'|----|----|----|----|----|----|----|----|----|----|'
 | b'***************************************************'
 | b''
 | b':: image does not contain valid values'
 +---------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/vito/icor/src/icor/context.py", line 497, in invoke_ac_runner_check
    p = subprocess.check_output(cmd, shell=True,  stderr=subprocess.STDOUT,env=env)
  File "/home/binbin/miniconda3/envs/icor/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/home/binbin/miniconda3/envs/icor/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconffu58njrc.conf' returned non-zero exit status 255.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/vito/icor/src/icor.py", line 45, in process_product
    icor.sentinel2.process_saf(context, product,working_folder)
  File "/opt/vito/icor/src/icor/sentinel2.py", line 38, in process_saf
    sentinel2_granule(granule,context,dir)
  File "/opt/vito/icor/src/icor/sentinel2.py", line 643, in sentinel2_granule
    "ERROR : AOT values cannot be retrieved from the image - please use default aot value"
  File "/opt/vito/icor/src/icor/context.py", line 515, in invoke_ac_runner_check
    raise Exception(comment)
Exception: process failed ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconffu58njrc.conf
in stage : Estimate the Aerosol Optical Thickness (AOT)
ERROR : AOT values cannot be retrieved from the image - please use default aot value

======================================================================
 52% Remove intermediate files from filesystem
======================================================================
delete tmp files failed
Traceback (most recent call last):
  File "/opt/vito/icor/src/icor.py", line 76, in process_product
    context.remove_tmp_files(working_folder,keep_tmp, output_folder)
  File "/opt/vito/icor/src/icor/context.py", line 798, in remove_tmp_files
    self.remove_dimap_subfolder()
  File "/opt/vito/icor/src/icor/context.py", line 703, in remove_dimap_subfolder
    os.chdir(dimap_vector_folder)
FileNotFoundError: [Errno 2] No such file or directory: '/home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/dimap_file.data/vector_data'

could not remove tmpfiles from /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc. Please check manually.
------ERROR-------
process failed due to error : 
process failed ac_runner /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/atcconffu58njrc.conf
in stage : Estimate the Aerosol Optical Thickness (AOT)
ERROR : AOT values cannot be retrieved from the image - please use default aot value
------ERROR-------
process end
iCOR Atmospheric correction done for product : /home/binbin/Append/icortry/tmp/icor_kkwxwdvc_proc/S2A_OPER_MSI_L1C_TL_MTI__20181008T205913_A017211_T17RNK_N02_06/S2A_OPER_MSI_L1C_TL_MTI__20181008T205913_A017211_T17RNK_N02_06

saponarom · March 1, 2022, 12:25pm

Hi everyone,
I have the exact same problem with the Estimate AOT value parameter, if checked, with S2 images of February 2022. In particular, the iCOR algorithm continues to work with images of the same scenario from 8 January and 13 January, while the same bbrauzzi’s error occurs with images of 2 February and 22 February. It should be noted that the above images have a cloud cover percentage of less than 30% and no portions of water, snow, desert, etc. are framed.

Looking through the news on Scihub.copernicus.eu I came across this communication: Open Access Hub

I read that Sen2Cor has been updated to the new Processing Baseline implementation of 25 January 2022. Do you think iCOR needs to be updated in the same way?

Thanks in advance!
Mirko

LH002 · June 19, 2022, 4:39am

Hello, I also encountered this problem when processing the data of sentinel2 in 2022. Do you have a solution now?

marpet · June 20, 2022, 1:40pm

Hi saponarom and LH002,

probably icor needs an update fir the S2 data of PB4.0.0.
The icor team is looking here sometimes for new questions but not on a regular basis.
It might be better if you contact VITO directly.
They have a contact formular:
Contact Us | VITO Remote Sensing

LH002 · June 21, 2022, 2:25am

Thank you!
Regarding ICOR module’s handling of sentinel-2 exceptions, I tried the following solutions to make the data run normally:

adjust the cloud mask area threshold suitable for the image according to the cloud Mask Algorithm in the manual;
adjust the threshold of water body inundation area;

The reason for abnormal atmospheric correction processing may be that the whole scene image is recognized as cloud when the cloud mask is used and the whole image is recognized as a water body

LH002 · June 21, 2022, 3:10am

Maybe you can try my method to adjust the cloud recognition threshold and water recognition threshold.

After I adjust the threshold parameters, ICOR can estimate AOT products.

mmoerk · January 13, 2023, 6:07pm

Hi LH002, I am also working with processing some Sentinel-2 imagery via iCOR and am running into the same problems you previously had. Adjusting the water mark detection threshold, alongside the cloud mask area manually does allow iCOR to run, however doing so with your inputs returns non-logical values. May I ask how you decided upon your manual input values for these parameters?

Thanks for your help!

LWX5990910 · February 8, 2023, 9:26am

hi，I’m having the same problem as you in dealing with sentinel-2, can I ask you if I solved this problem?

LWX5990910 · February 8, 2023, 10:04am

HI， thank you very much for the parameter settings, let me complete the Sentinel-2 image atmospheric correction, here is the result I got, I think this is not an ideal situation, normally after the near-infrared band the water reflectance value should tend to 0, but this result shows the opposite trend.

LH002 · February 9, 2023, 2:15am

I use L1 data for cloud mask detection according to the cloud detection algorithm in the iCOR document description; Through L1 data, the water threshold is re-determined. The threshold coefficient determined until the detection result is consistent with the image result.
Maybe you can try!

LH002 · February 9, 2023, 2:17am

Can you try to adjust the relevant threshold? It may be that the threshold is not suitable for the scene you are processing.

LWX5990910 · February 16, 2023, 3:19am

Hi, thank you very much, you are right, I solved this problem.