Sen2Cor batch processing

Hi,
I’m interested in batch-processing several Sentinel-2 images on a 8 core, 32 GB RAM, 4 GB video, Windows 10 desktop. I got started with Sen2Cor and it takes me 30-45 minutes per tile.

Wondering if anyone has experience batch-processing in Sen2Cor and has created a workflow that I can use or modify.

What has been your experience so far with batch processing? What computer specs will improve it greatly?

Found this paper on applying Sen2Cor on the EODC framework “Data Service Platform for Sentinel-2 Surface Reflectance and Value-Added Products: System Use and Examples” (Remote Sensing, 2016, 8, 938). They use high specs and it still takes per granule 37 minutes for atmospheric correction and 15 minutes for value-added product generation.

1 Like

Use parallel processes / multiproces. I have a similar configuration. Configure 4 parallel processes (4 cores). If more processes are involved, it is possible to make a mistake. The process for 12 tiles takes 120 minutes.

1 Like

Can you provide specific steps on how I can enable and configure parallel processes?
It took 43 minutes on my desktop for
S2A_MSIL1C_20171004T184221_N0205_R070_T11UQS_20171004T184527.SAFE

C:\Users\EnviMatix-AMDFX.snap\auxdata\Sen2Cor-2.4.0-win64\Lib\site-packages\sen2cor\cfg\L2A_GIPP.xml -> C:\Users\EnviMatix-AMDFX\Documents\sen2cor\cfg\L2A_GIPP.xml
1 File(s) copied
Sentinel-2 Level 2A Processor (Sen2Cor), 2.4.0, created: 2017.06.05 started …
selected resolution is 10 m.
Progress[%]: 0.07 : PID-18480, 20 m resolution must be processed first, elapsed time[s]: 1.570
Progress[%]: 0.08 : PID-18480, L2A_ProcessTile: processing with resolution 20 m, elapsed time[s]: 0.398
Progress[%]: 0.08 : PID-18480, L2A_ProcessTile: start of pre processing, elapsed time[s]: 0.000
Progress[%]: 0.09 : PID-18480, L2A_Tables: start import, elapsed time[s]: 0.156
Progress[%]: 0.64 : PID-18480, L2A_Tables: band B01 imported, elapsed time[s]: 13.172
Progress[%]: 2.49 : PID-18480, L2A_Tables: band B02 imported, elapsed time[s]: 44.461
Progress[%]: 4.35 : PID-18480, L2A_Tables: band B03 imported, elapsed time[s]: 44.531
Progress[%]: 6.20 : PID-18480, L2A_Tables: band B04 imported, elapsed time[s]: 44.399
Progress[%]: 6.74 : PID-18480, L2A_Tables: band B05 imported, elapsed time[s]: 13.095
Progress[%]: 7.28 : PID-18480, L2A_Tables: band B06 imported, elapsed time[s]: 12.889
Progress[%]: 7.82 : PID-18480, L2A_Tables: band B07 imported, elapsed time[s]: 13.063
Progress[%]: 8.36 : PID-18480, L2A_Tables: band B8A imported, elapsed time[s]: 12.816
Progress[%]: 8.90 : PID-18480, L2A_Tables: band B09 imported, elapsed time[s]: 13.075
Progress[%]: 9.38 : PID-18480, L2A_Tables: band B10 imported, elapsed time[s]: 11.565
Progress[%]: 9.91 : PID-18480, L2A_Tables: band B11 imported, elapsed time[s]: 12.623
Progress[%]: 10.41 : PID-18480, L2A_Tables: band B12 imported, elapsed time[s]: 12.046
Progress[%]: 10.41 : PID-18480, L2A_ProcessTile: start of Scene Classification, elapsed time[s]: 0.014
Progress[%]: 10.49 : PID-18480, Pre process , elapsed time[s]: 1.898
Progress[%]: 10.93 : PID-18480, L2A_SC init , elapsed time[s]: 10.573
Progress[%]: 11.05 : PID-18480, L2A_CSND_1_1 , elapsed time[s]: 2.738
Progress[%]: 11.16 : PID-18480, L2A_CSND_1_2 , elapsed time[s]: 2.860
Progress[%]: 11.19 : PID-18480, L2A_CSND_2_0 , elapsed time[s]: 0.605
Progress[%]: 11.30 : PID-18480, L2A_CSND_2_1 , elapsed time[s]: 2.675
Progress[%]: 11.40 : PID-18480, L2A_CSND_2_1_2, elapsed time[s]: 2.300
Progress[%]: 11.45 : PID-18480, L2A_CSND_2_2 , elapsed time[s]: 1.378
Progress[%]: 11.51 : PID-18480, L2A_CSND_2_3 , elapsed time[s]: 1.382
Progress[%]: 11.61 : PID-18480, L2A_CSND_2_4 , elapsed time[s]: 2.463
Progress[%]: 11.72 : PID-18480, L2A_CSND_2_5 , elapsed time[s]: 2.653
Progress[%]: 11.90 : PID-18480, L2A_CSND_3 , elapsed time[s]: 4.105
Progress[%]: 12.04 : PID-18480, L2A_CSND_5_1 , elapsed time[s]: 3.446
Progress[%]: 12.30 : PID-18480, L2A_CSND_5_2 , elapsed time[s]: 6.307
Progress[%]: 12.49 : PID-18480, L2A_CSND_6 , elapsed time[s]: 4.415
Progress[%]: 12.62 : PID-18480, L2A_CSND_6_2 , elapsed time[s]: 3.254
Progress[%]: 12.77 : PID-18480, L2A_CSND_7 , elapsed time[s]: 3.607
Progress[%]: 14.86 : PID-18480, L2A_SHD , elapsed time[s]: 50.191
Progress[%]: 15.02 : PID-18480, DV recovery , elapsed time[s]: 3.651
Progress[%]: 15.18 : PID-18480, WP recovery , elapsed time[s]: 3.980
Progress[%]: 15.32 : PID-18480, Snow recovery , elapsed time[s]: 3.227
Progress[%]: 15.40 : PID-18480, Soil recovery , elapsed time[s]: 2.013
Progress[%]: 15.87 : PID-18480, Post process , elapsed time[s]: 11.279
Progress[%]: 15.87 : PID-18480, L2A_ProcessTile: start of Atmospheric Correction, elapsed time[s]: 0.000
Progress[%]: 16.00 : PID-18480, L2A_AtmCorr: end of calculation terrain maps, elapsed time[s]: 3.177
Progress[%]: 16.00 : PID-18480, L2A_AtmCorr: start of AOT retrieval at 550nm, elapsed time[s]: 0.002
Progress[%]: 16.62 : PID-18480, L2A_AtmCorr: end of internal classification, elapsed time[s]: 14.769
Progress[%]: 16.65 : PID-18480, L2A_AtmCorr: end of interpolation LUTs, elapsed time[s]: 0.834
Progress[%]: 16.65 : PID-18480, L2A_AtmCorr: end retrieving reference pixels for dark areas, elapsed time[s]: 0.016
Progress[%]: 18.03 : PID-18480, L2A_AtmCorr: end of check for dense bright vegetation pixels, elapsed time[s]: 33.108
Progress[%]: 18.34 : PID-18480, L2A_AtmCorr: end of receiving atmospheric functions for all altitudes and visibilities, elapsed time[s]: 7.277
Progress[%]: 18.34 : PID-18480, L2A_AtmCorr: end of AOT retrieval at 550nm, elapsed time[s]: 0.006
Progress[%]: 18.34 : PID-18480, L2A_AtmCorr: start of water vapour retrieval, elapsed time[s]: 0.006
Progress[%]: 18.59 : PID-18480, L2A_AtmCorr: end of water vapour retrieval preparation, elapsed time[s]: 6.128
Progress[%]: 31.97 : PID-18480, L2A_AtmCorr: end of water vapour retrieval, elapsed time[s]: 321.178
Progress[%]: 31.97 : PID-18480, L2A_AtmCorr: preparation of surface reflectance retrieval, elapsed time[s]: 0.000
Progress[%]: 32.26 : PID-18480, L2A_AtmCorr: end of surface reflectance retrieval preparation, elapsed time[s]: 6.813
Progress[%]: 42.62 : PID-18480, L2A_AtmCorr: end of surface reflectance retrieval, elapsed time[s]: 248.683
Progress[%]: 47.00 : PID-18480, L2A_AtmCorr: end of rho retrieval step 2, elapsed time[s]: 105.239
Progress[%]: 47.01 : PID-18480, L2A_ProcessTile: start of post processing, elapsed time[s]: 0.078
Progress[%]: 47.01 : PID-18480, L2A_Tables: start export, elapsed time[s]: 0.000
Progress[%]: 47.66 : PID-18480, L2A_Tables: band B02 exported, elapsed time[s]: 15.735
Progress[%]: 48.33 : PID-18480, L2A_Tables: band B03 exported, elapsed time[s]: 15.983
Progress[%]: 48.98 : PID-18480, L2A_Tables: band B04 exported, elapsed time[s]: 15.595
Progress[%]: 49.61 : PID-18480, L2A_Tables: band B05 exported, elapsed time[s]: 15.140
Progress[%]: 50.23 : PID-18480, L2A_Tables: band B06 exported, elapsed time[s]: 14.847
Progress[%]: 50.84 : PID-18480, L2A_Tables: band B07 exported, elapsed time[s]: 14.669
Progress[%]: 51.44 : PID-18480, L2A_Tables: band B8A exported, elapsed time[s]: 14.423
Progress[%]: 51.95 : PID-18480, L2A_Tables: band B11 exported, elapsed time[s]: 12.185
Progress[%]: 52.44 : PID-18480, L2A_Tables: band B12 exported, elapsed time[s]: 11.771
Progress[%]: 52.56 : PID-18480, L2A_Tables: band SCL exported, elapsed time[s]: 2.886
Progress[%]: 52.82 : PID-18480, L2A_Tables: band SNW exported, elapsed time[s]: 6.265
Progress[%]: 53.14 : PID-18480, L2A_Tables: band CLD exported, elapsed time[s]: 7.649
Progress[%]: 53.20 : PID-18480, L2A_Tables: band AOT exported, elapsed time[s]: 1.495
Progress[%]: 53.46 : PID-18480, L2A_Tables: band WVP exported, elapsed time[s]: 6.294
Progress[%]: 53.51 : PID-18480, L2A_Tables: band VIS exported, elapsed time[s]: 1.219
Progress[%]: 53.66 : PID-18480, L2A_Tables: band PVI exported, elapsed time[s]: 3.532
Progress[%]: 54.62 : PID-18480, L2A_Tables: band TCI exported, elapsed time[s]: 23.046
Progress[%]: 54.64 : PID-18480, L2A_Tables: stop export, elapsed time[s]: 0.344
Progress[%]: 54.67 : PID-18480, L2A_ProcessTile: processing with resolution 10 m, elapsed time[s]: 0.727
Progress[%]: 54.67 : PID-18480, L2A_ProcessTile: start of pre processing, elapsed time[s]: 0.016
Progress[%]: 54.67 : PID-18480, L2A_Tables: start import, elapsed time[s]: 0.078
Progress[%]: 56.77 : PID-18480, L2A_Tables: band B02 imported, elapsed time[s]: 50.365
Progress[%]: 58.89 : PID-18480, L2A_Tables: band B03 imported, elapsed time[s]: 50.824
Progress[%]: 61.01 : PID-18480, L2A_Tables: band B04 imported, elapsed time[s]: 50.944
Progress[%]: 63.12 : PID-18480, L2A_Tables: band B08 imported, elapsed time[s]: 50.592
Progress[%]: 64.56 : PID-18480, L2A_Tables: band AOT imported, elapsed time[s]: 34.517
Progress[%]: 66.44 : PID-18480, L2A_Tables: band WVP imported, elapsed time[s]: 45.239
Progress[%]: 68.03 : PID-18480, L2A_Tables: band VIS imported, elapsed time[s]: 38.167
Progress[%]: 69.74 : PID-18480, L2A_Tables: band SCL imported, elapsed time[s]: 41.049
Progress[%]: 69.74 : PID-18480, L2A_ProcessTile: start of Atmospheric Correction, elapsed time[s]: 0.047
Progress[%]: 70.30 : PID-18480, L2A_AtmCorr: end of calculation terrain maps, elapsed time[s]: 13.408
Progress[%]: 70.30 : PID-18480, L2A_AtmCorr: start of resampling visibility for 10 m resolution, elapsed time[s]: 0.000
Progress[%]: 70.51 : PID-18480, L2A_AtmCorr: end of resampling visibility for 10 m resolution, elapsed time[s]: 5.091
Progress[%]: 70.75 : PID-18480, L2A_AtmCorr: end of AOT retrieval at 550nm, elapsed time[s]: 5.655
Progress[%]: 70.75 : PID-18480, L2A_AtmCorr: preparation of surface reflectance retrieval, elapsed time[s]: 0.015
Progress[%]: 70.98 : PID-18480, L2A_AtmCorr: end of surface reflectance retrieval preparation, elapsed time[s]: 5.470
Progress[%]: 84.34 : PID-18480, L2A_AtmCorr: end of surface reflectance retrieval, elapsed time[s]: 320.640
Progress[%]: 90.72 : PID-18480, L2A_AtmCorr: end of rho retrieval step 2, elapsed time[s]: 153.063
Progress[%]: 90.72 : PID-18480, L2A_ProcessTile: start of post processing, elapsed time[s]: 0.156
Progress[%]: 90.72 : PID-18480, L2A_Tables: start export, elapsed time[s]: 0.000
Progress[%]: 93.30 : PID-18480, L2A_Tables: band B02 exported, elapsed time[s]: 61.797
Progress[%]: 95.91 : PID-18480, L2A_Tables: band B03 exported, elapsed time[s]: 62.797
Progress[%]: 98.10 : PID-18480, L2A_Tables: band B04 exported, elapsed time[s]: 61.422
Progress[%]: 99.63 : PID-18480, L2A_Tables: band B08 exported, elapsed time[s]: 59.149
Progress[%]: 99.59 : PID-18480, L2A_Tables: band AOT exported, elapsed time[s]: 7.834
Progress[%]: 99.63 : PID-18480, L2A_Tables: band WVP exported, elapsed time[s]: 18.641
Progress[%]: 99.65 : PID-18480, L2A_Tables: band TCI exported, elapsed time[s]: 107.703
Progress[%]: 99.31 : PID-18480, L2A_Tables: stop export, elapsed time[s]: 0.719
Progress[%]: 100.00 : Application terminated successfully.
Finished tool execution in 2574 seconds

@davorinbajic Can you provide specific steps on how I can enable and configure parallel processes?

Hallo envimatix,

I am using a Linux shell script for batch processing with Sen2Cor. I put all L1C-files to be processed into one directory and then parse through the file list of this directory calling Sen2Cor for each product. Parallel processing gives no advantage for me because I am processing only single-granule products. However, it should be possible to organize several separate jobs running in parallel to reduce the processing time for the whole data set.

Do you mind sharing your Linux shell script for batch processing with Sen2Cor?

Hi,

Is it possible to share the script with me? I also want to batch processing L1C files for atmospheric correction in Linux.

Thank you!

Han