Bulk processing for Sentinel-3


#1

Hi,

I am trying to adapt the unix script proposed in this guide (originally desing to work with S2 data) to work with Sentinel-3. The first one is a generic issue:

  • When runing the script in Linux, I have found that depending on your permissions (that you can check with ls -l path/to/the/folder/where/your/script/is ) you will need to precede the code with bash
  • For some reason, the shell does not interpret properly the empty lines. I have fixed that using :

tr -d '\015' </path/to/the/script/processDataset.bash >/path/to/the/script/processDataset.basht_V2.bash

  • My question/problem is now how to import the Sentinel-3 images properly. In line 80, this is used for Sentinel-2
    for F in $(ls -1 "${sourceDirectory}"/S2*.SAFE); do

I have tried several variations likeS3*.SEN3 or S3*.SEN3/xfdumanifest.xml or S3* but of course it is not working. Any idea how to set this properly to make to script work with bulk processing of S3 data? Any experience on that?

When using the option S3*.SEN3 (which follows the same approach as the original one for Sentinel2 - S2*.SAFE), received the following error:

"/shared/Downloads/Chipre/June/S3A_OL_1_EFR____20170605T080114_20170605T080414_20170606T121925_0179_018_249_2340_LN1_O_NT_002.SEN3/:": No such file or directory

and then a long list of error s like this

/shared/Downloads/Chipre/processDataset_V4.bash: line 84: : command not found
/shared/Downloads/Chipre/processDataset_V4.bash: line 83: "/home/rus/Oa01_radiance.nc": No such file or directory
/shared/Downloads/Chipre/processDataset_V4.bash: line 84: : command not found
/shared/Downloads/Chipre/processDataset_V4.bash: line 83: "/home/rus/Oa02_radiance.nc": No such file or directory

It is strange since its looking for the files of the image in a different path (home/rus/…)

M


Subset batch process problem
#2

I think you second try should have worked.
The for loop header should look like:

for F in $(ls -1 "${sourceDirectory}"/S3*.SEN3/xfdumanifest.xml); do

In the loop you can add a

echo ${F} # this should print just the file name

and

echo ${sourceFile} # this should print the absolute path

The you can see the path of the files and if they are correctly evaluated.


#3

@marpet

I have check the echo ${F} to see the path it is creating and it is correct although it claims no such file or directory
I would say the problem is in line 83, where the β€œ${sourceFile}” variable is combined with the rest. I think it is not interpreting the \"${sourceFile}\" as a variable assignment. Might be that the space are not escaped?

I have replicated the original exercise, using same graph, sentine-2 images, etc. and I get the same error.


#4

I must confess that I’m not a unix script expert. I have written the initial windows script and a colleague wrote the unix following the windows script. I’ve tried to maintain the unix script but it could be that I made one or two mistakes. But usually I test the scripts after changing them.
Later this day I will have closer look.


#5

If I run the script directly instead of building the varialbe it works. I’ll have a look to how this part of the script creates the command and let you know too

gpt /path/to/graph/myGraph_v3_for_GPT_v2_2.xml -e -p /path/to/variables.properties -t /path/to/outputdirectory/ /path/to/sentinel3or2image/S3A_OL_1_EFR____20170602T073847_20170602T074147_20170603T115248_0179_018_206_2340_LN1_O_NT_002.SEN3/xfdumanifest.xml


#6

I have modified two main things in the script:

  • line 36 -> remove function (if not errors may appear)

  • line 83:-> reorganize the variable cottation

New version:
procCmd="${gptPath}" "${graphXmlPath}" "/-e -p /" "${parameterFilePath}" "/ -t /" "${targetFile}" "${sourceFile}"

Old Version:
procCmd="\"${gptPath}\" \"${graphXmlPath}\" -e -p \"${parameterFilePath}\" -t \"${targetFile}\"" \"${sourceFile}\"

This seems to solve the issue but I am now facing some Linux-based problems to run the script properly. A difference that I have found with the original guide is that I have to specify the full path the the parameters, not only their names. After running (with absolute path), it starts to complain about the graph

/shared/Downloads/Chipre/myGraph_v3_for_GPT_v2_2.xml: line 1: syntax error near unexpected token `newline'
/shared/Downloads/Chipre/myGraph_v3_for_GPT_v2_2.xml: line 1: `<graph id="Graph">'
/shared/Downloads/Chipre/processDataset_V4.bash: line 84: : command not found

But if I use chmod +x before the script the problem is that

`chmod +x /path/to/processDataset_V4.bash /path/to/myGraph_v3_for_GPT_v2_2.xml /path/to/variables.properties β€œ/input/iamges/path” β€œ/output/targets/path” ndvi

the error is that it does not recognize the ndvi as the variable that is used in the scritp to add to the target products

chmod: cannot access 'ndvi': No such file or directory

Any Linux experienced user that can hava a look to this?

M


#7

Hi!

I am facing an identical error. Did someone came up with the solution for the unix version of the script for S3?


#8

Today I had a look at the script. One problem was that it contained windows line endings and a quote was wrongly set.
Also I did some simplification. For a test run it worked, but maybe there are still some edge cases where it might fail.
Please, let us know if you experience any issue.


#9

Hi,
I am trying to learn how to process Sentinel-3 (batch), and the first thing I am doing is process Level-1 images with c2rcc. I am using windows.
This is how I configured my processDataset.bat file:

::@echo off
:: enable delayed expansion - used to resolve variable in loop
:: variable has to be used with β€˜!’ instead of β€˜%’
setlocal ENABLEDELAYEDEXPANSION
::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::
:: adapt this path to your needs
set gptPath=β€œC:\Program Files\snap\bin\gpt.exe”
::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling
::::::::::::::::::::::::::::::::::::::::::::
:: first parameter is a path to the graph xml
set graphXmlPath=β€œD:\IMAGES\C2RCC\c2rcc_olci.xml”

:: second parameter is a path to a parameter file
:: set parameterFilePath=β€œD:\IMAGES\C2RCC\resample_20m.properties”
I do not want to change any parameter now. Should I just skip this line?

:: use third parameter for path to source products
set sourceDirectory=D:\IMAGES\SEN_L1\BC_coast\test
:: if sourceDirectory ends with β€˜β€™ remove it
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%

:: use third parameter for path to target products
set targetDirectory=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch
:: if targetDirectory ends with β€˜β€™ remove it
if %targetDirectory:~-1%==\ set targetDirectory=%targetDirectory:~0,-1%

:: the fourth parameter is a file prefix for the target product name,
:: typically indicating the type of processing
set targetFilePrefix=C2RCC

:: Create the target directory
md D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch\test
::::::::::::::::::::::::::::::::::::::::::::
:: Main processing
::::::::::::::::::::::::::::::::::::::::::::
:: double β€˜%’ in batch file and only a single β€˜%’ on command line
:: β€˜/D’ is for directories like Sentinel data. Remove β€˜/D’ when you open files.
for /D /R %sourceDirectory% %%F in (S3*.SEN3/xfdumanifest.xml) do (
echo.
:: β€˜~fF’ means abolute path of β€˜F’
set sourceFile=%%~fF
echo Processing !sourceFile!
:: β€˜~nF’ means filename without extension of β€˜F’
set targetFile=%targetDirectory%%targetFilePrefix%_%%~nF.dim
set procCmd=%gptPath% %graphXmlPath% -e -p %parameterFilePath% -t β€œ!targetFile!” β€œ!sourceFile!”
call !procCmd!
)

What do I still need to edit to make it work? When I write in the command line prompt:
D:\IMAGES\C2RCC\processDataset.bat c2rcc_olci.xml D:\IMAGES\SEN_L1\BC_coast\test D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch C2RCC
This is what I get and nothing happens:

setlocal ENABLEDELAYEDEXPANSION
set gptPath=β€œC:\Program Files\snap\bin\gpt.exe”
set graphXmlPath=\IMAGES\C2RCC\c2rcc_olci.xml
set sourceDirectory=D:\IMAGES\SEN_L1\BC_coast\test
if t == \ set sourceDirectory=D:\IMAGES\SEN_L1\BC_coast\tes
set targetDirectory=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch
if h == \ set targetDirectory=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batc
set targetFilePrefix=C2RCC
md D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch\test
for / %F in (S3*.SEN3/xfdumanifest.xml) do (
echo.
set sourceFile=%~fF
echo Processing !sourceFile!
set targetFile=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch\C2RCC_%~nF.dim
set procCmd=β€œC:\Program Files\snap\bin\gpt.exe” \IMAGES\C2RCC\c2rcc_olci.xml -e -p -t β€œ!targetFile!” β€œ!sourceFile!”
call !procCmd!
)

I would appreciate any help on how to start this kind of work.

Thank you,