SenCor 2.3.0 parallel processing on the new format 14.2

As we know, the new SAFE file format of Sentinel-2 has only one L1C tile in each Safe file.
So how can I set the sen2cor to let it process the data using multi cores in parallel ?

Thanks!

Hello,
I use R software. My friend wrote for me short script in order to make atmospheric corrections parallel for X tiles.

Regards

just restart it again with a different filename.

Hallo,

running Sen2Cor from command line you can configure in the L2A_GIPP.xml file the number or processes to run in parallel.

<?xml version="1.0" encoding="UTF-8"?>

<Level-2A_Ground_Image_Processing_Parameter xmlns:xsi=β€œhttp://www.w3.org/2001/XMLSchema-instance” xsi:noNamespaceSchemaLocation=β€œL2A_GIPP.xsd”>
<Common_Section>
<Log_Level>INFO</Log_Level>

<Nr_Processes>2</Nr_Processes>

…

bdpg

@bdpg

can you elaborate how to use this setting?
The manual states that the number of processes is set automatically to the given architecture. But when I start one sen2cor process my CPU (and RAM) are only used to a very low capacity.

Hallo unnic,

You can set the number of processors to be used for parallel processing manually in the user GIPP-file or set at the same location the option of automatic determination the number of available processors. If you are in doubt that processes doesn’t run in parallel, then set the number of processors manually and check if anything is changing. Note that you may not set to use more processors than your computer has. Note also, that parallel processing allows processing multiple tiles of a larger product parallel. Processing a single tile is a sequential flow and doesn’t works parallel.

Kind regards
Bringfried Pflug

β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”
Deutsches Zentrum fΓΌr Luft- und Raumfahrt (DLR) - German Aerospace Center | Earth Observation Center | Remote Sensing Technology Institute | Photogrammetry and Image Analysis | Team Marine Remote Sensing | Rutherfordstrasse 2 | 12489 Berlin | Germany

Bringfried Pflug
Telephone +49 30 67055-655 | Telefax +49 30 67055-642 | bringfried.pflug@dlr.demailto:bringfried.pflug@dlr.de www.DLR.de/eochttp://www.dlr.de/eoc

Ok, so this mean that in the way the data is provided now (single tile format) this feature cannot be used.

Yes, parallel processing cannot be used processing the new single-tile products.

bdpg

In case you are running under Linux, have a look at GNU-parallel.
I am using it successfully to run multiple instances of sen2cor in parallel, even distributed across different machines.
Really all it takes is a list of files to process.
Here’s a simple example for a single machine with four cores, whereby FILELIST.txt lists the .SAFE directory names:

parallel -j4 'L2A_Process {}' :::: FILELIST.txt

Watch out for memory consumption, though. At least one time during processing sen2cor loads the whole granule into memory.

1 Like

That works under any OS (it’s just calling L2A_Process on different SAFE files which you can do on other systems as well).

:slight_smile: that was not the point. The point was to give an actual example of how to achieve this.
Of course, it can be done on any platform in a million different ways and as I just found out, some people even use GNU-parallel on windows (via cygwin).