Reducing sen2cor processing time

Hello,

is there a way to reduce the processing time of sen2cor? It would be interesting for me to limit the L2A_Process on selected granules, or omitting 60 m and 20 m calculation…

I’m thankful for any information.

1 Like

Omitting 60 and 20m would also be interesting for me. Regarding processing only certain granules:

Answer

1 Like

10 m requires the 20 m processing, as the Water vapour and AOT requires the other bands < 10 and is an input for 10 m processing. But processing of selected tiles and processing 20m without 60 is now possible. Is a feature of the release 2.2.1 which has been just released.

1 Like

How much time is it taking for the processing of one scene?
I had one image processed in about 8 hours. Since then, it is taking more than 24 hours per scene. Sometimes I am leaving the computer to process for several days that I am just stopping the processing without obtaining results. Is there something wrong? Any thoughts?

1 Like

24 Hrs, even 8 Hrs is much to much. Normally you can expect the processing of a single tile in about 1.5 Hrs for all three resolutions. This is however dependent on the architecture of your Machine. With a standard Processor and ~ 8 GB of RAM this is the processing time to be expected. If you have considerable less memory, the results are very probable explained by swapping activities of your harddisk, as the processing of 10000 x 10000 pixel images during atmospheric correction needs ecessive amount of memory.

Cheers,
Uwe

Thank you. But, this should not be the problem (RAM 16 GB - and relatively new computer). Now, I am processing an image. Im reaching soon 36 hours of processing and it gives me now on the Command Prompt “42.64 : PID-4124”. I imagine that is 42.62% of the process???
I’m leaving it to see what will happen, but I think there is a problem somewhere else, if you can think of another possible reason, I would greatly appreciate it.

p.s. I processed the same image by iCor in less than an hour.

I have been processing since yesterday (24h+ ) for now. I have 56 GB of spare RAM and CPU usage (12 threads) is around 9-10 %. I am using sen2cor stand alone installation, v. 2.4.0. Progress [%] was at the morning around 13 % (after around 18 hrs of processing), then jumped suddenly to 97 % and has now been varying the past 6 hours between 97 and 99 %, back and forth. Surely this can’t be how it is supposed to be?! Any way to correct this?

1 Like

can you give some information about system?

os ? 32Bit or 64?

which installer of sen2cor? standalone or anaconda?

Win7, 64 Bit (Sp 1.). Stand alone installer 2.4.0, downloaded and installed the very same day. It is still processing, the last messages are;

Progress[%]: 97.83 : PID-304, L2A_AtmCorr: start of water vapour retrieval, elapsed time[s]: 0.000
Progress[%]: 97.84 : PID-304, L2A_AtmCorr: end of water vapour retrieval preparation, elapsed time[s]: 5.210
Progress[%]: 99.90 : PID-304, L2A_AtmCorr: end of water vapour retrieval, elapsed time[s]: 58926.361
Progress[%]: 99.53 : PID-304, L2A_AtmCorr: preparation of surface reflectance retrieval, elapsed time[s]: 0.000
Progress[%]: 99.17 : PID-304, L2A_AtmCorr: end of surface reflectance retrieval preparation, elapsed time[s]: 3.647
Progress[%]: 99.21 : PID-304, L2A_AtmCorr: end of surface reflectance retrieval, elapsed time[s]: 189.163
Progress[%]: 99.03 : PID-304, L2A_AtmCorr: end of rho retrieval step 2, elapsed time[s]: 86.170
Progress[%]: 98.65 : PID-304, L2A_ProcessTile: start of post processing, elapsed time[s]: 0.062
Progress[%]: 98.28 : PID-304, L2A_Tables: start export, elapsed time[s]: 0.000
Progress[%]: 97.93 : PID-304, L2A_Tables: band B02 exported, elapsed time[s]: 11.120
Progress[%]: 97.96 : PID-304, L2A_Tables: band B03 exported, elapsed time[s]: 11.087

CPU usage and RAM remains the same as yesterday.

What exactly does the water vapor retrieval do, if it takes that long?

Ps. The image I am trying to process is, downloaded from scihub; S2A_USER_PRD_MSIL2A_PDMC_20160817T180350_R008_V20160816T104022_20160816T104025.SAFE

Hi @reimari,

I have had a look to your product (I think it is S2A_OPER_PRD_MSIL1C_PDMC_20160817T180350_R008_V20160816T104022_20160816T104025.SAFE) and it contains 18 granules. Sen2cor computes every granule independently and the commented processing time (around 1.5h) refers to the processing time for one granule.

Yes, you are correct, the input of course is L1C, and what I am hoping to achieve is the L2A :slight_smile:
This is my first Sentinel 2 image, so I didn’t know that there are multiple granules. Is there anyway to subset the granules?

Anyway, some parallel processing option would speed up the processing quite much. At least in my case there’s plenty of CPU and RAM unused. For example it would be great if the program could split each granule to one CPU thread.

EDIT: It is still processing (so ~48 hrs of processing now behind, which makes at least 2.5+ hrs / granule).

Hello, it is possible to process per Tile as shown in the Sen2Cor Manual http://step.esa.int/thirdparties/sen2cor/2.4.0/Sen2Cor_240_Documenation_PDF/S2-PDGS-MPC-L2A-SUM-V2.4.0.pdf . This way is much faster

1 Like

Just returned from Christmas holiday and it was still processing. Now cancelled. That must somehow be just never-ending loop which it was processing?

I’ll now try to process a single tile at once. Thanks for the tip!

EDIT:

Single granule processing went smoothly. No problems there. No idea what was wrong when processing the whole file.

I am glad I could help.

1 Like

How exactcly does it mean “Single granule processing”? Can you do it from SNAP?

I have tried to process one L1C image but I have the above commented problem, it takes forever and goes from 97% to 99% back and forth.

The image is from 2016 and the snap version is 2.5.5.

Hello @candelas94 Jaime

Prior to December 2016, Sentinel-2 products were available in ‘slice’ format. This was a collection of a number of Tiles, which mean the resulting product could be 7 - 11GB in size. In my experience, this made them unwieldy. After mid-December 2016, products were released in single Tile format (A Tile is also a Granule (Sentinel-2 - Data Products - Sentinel Handbook - Sentinel Online), with a shorter naming convention that was introduced because of issues in Windows 255 character limit (User Guides - Sentinel-2 MSI - Naming Convention - Sentinel Online - Sentinel Online). These products are around 800MB (L1C) and ~ 1.0 GB (L2A).

From my side. I think it would be the better if you can update to the most recent version of SNAP (9.0) SNAP Download – STEP

Cheers

Jan

Jan Jackson

OPT-MPC S2 Technical Manager

Thank you, Jan, for the clarification. It is good to know how things have evolved to understand them better :slight_smile:

I am using SNAP 9.0 already but thank you for the suggestion. I will have to look for other related questions and see if I find out what is happening. I had to stop the processing after ~38 hours of going from 97% to 99%.

Cheers

Jaime

Hello @candelas94 Jaime

Jeepers. That’s painful!

Have you tried Sen2Cor in standalone? i.e. outside SNAP? An L1C Tile takes about 20 minutes. for me. Sen2Cor v2.10 – STEP

Cheers

Jan

OPT-MPC S2 Technical Manager

Extreme processing slowdowns can indicate a hardware problem (disk nearly full or about to die, overheating due to fan failure or dust on cooling fins, …). You can use OS-dependent tools to check temperatures, disk health (S.M.A.R.T status), and CPU usage, but it is most useful if you can compare with previous runs when processing was faster. If your data are on an external drive there can be problems with the cables and ports. There are tools that do drive performance tests.

Hi. I tried to run it and it worked, or at least it produced a reproducible product that I can read in SNAP :sweat_smile:

1 Like