I am trying to adapt the unix script proposed in this guide (originally desing to work with S2 data) to work with Sentinel-3. The first one is a generic issue:
When runing the script in Linux, I have found that depending on your permissions (that you can check with ls -l path/to/the/folder/where/your/script/is ) you will need to precede the code with bash
For some reason, the shell does not interpret properly the empty lines. I have fixed that using :
My question/problem is now how to import the Sentinel-3 images properly. In line 80, this is used for Sentinel-2 for F in $(ls -1 "${sourceDirectory}"/S2*.SAFE); do
I have tried several variations like S3*.SEN3 or S3*.SEN3/xfdumanifest.xml or S3* but of course it is not working. Any idea how to set this properly to make to script work with bulk processing of S3 data? Any experience on that?
When using the option S3*.SEN3 (which follows the same approach as the original one for Sentinel2 - S2*.SAFE), received the following error:
"/shared/Downloads/Chipre/June/S3A_OL_1_EFR____20170605T080114_20170605T080414_20170606T121925_0179_018_249_2340_LN1_O_NT_002.SEN3/:": No such file or directory
and then a long list of error s like this
/shared/Downloads/Chipre/processDataset_V4.bash: line 84: : command not found
/shared/Downloads/Chipre/processDataset_V4.bash: line 83: "/home/rus/Oa01_radiance.nc": No such file or directory
/shared/Downloads/Chipre/processDataset_V4.bash: line 84: : command not found
/shared/Downloads/Chipre/processDataset_V4.bash: line 83: "/home/rus/Oa02_radiance.nc": No such file or directory
It is strange since its looking for the files of the image in a different path (home/rus/…)
I have check the echo ${F} to see the path it is creating and it is correct although it claims no such file or directory
I would say the problem is in line 83, where the “${sourceFile}” variable is combined with the rest. I think it is not interpreting the \"${sourceFile}\" as a variable assignment. Might be that the space are not escaped?
I have replicated the original exercise, using same graph, sentine-2 images, etc. and I get the same error.
I must confess that I’m not a unix script expert. I have written the initial windows script and a colleague wrote the unix following the windows script. I’ve tried to maintain the unix script but it could be that I made one or two mistakes. But usually I test the scripts after changing them.
Later this day I will have closer look.
If I run the script directly instead of building the varialbe it works. I’ll have a look to how this part of the script creates the command and let you know too
Old Version: procCmd="\"${gptPath}\" \"${graphXmlPath}\" -e -p \"${parameterFilePath}\" -t \"${targetFile}\"" \"${sourceFile}\"
This seems to solve the issue but I am now facing some Linux-based problems to run the script properly. A difference that I have found with the original guide is that I have to specify the full path the the parameters, not only their names. After running (with absolute path), it starts to complain about the graph
/shared/Downloads/Chipre/myGraph_v3_for_GPT_v2_2.xml: line 1: syntax error near unexpected token `newline'
/shared/Downloads/Chipre/myGraph_v3_for_GPT_v2_2.xml: line 1: `<graph id="Graph">'
/shared/Downloads/Chipre/processDataset_V4.bash: line 84: : command not found
But if I use chmod +x before the script the problem is that
Today I had a look at the script. One problem was that it contained windows line endings and a quote was wrongly set.
Also I did some simplification. For a test run it worked, but maybe there are still some edge cases where it might fail.
Please, let us know if you experience any issue.
Hi,
I am trying to learn how to process Sentinel-3 (batch), and the first thing I am doing is process Level-1 images with c2rcc. I am using windows.
This is how I configured my processDataset.bat file:
::@echo off
:: enable delayed expansion - used to resolve variable in loop
:: variable has to be used with ‘!’ instead of ‘%’
setlocal ENABLEDELAYEDEXPANSION
::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::
:: adapt this path to your needs
set gptPath=“C:\Program Files\snap\bin\gpt.exe”
::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling
::::::::::::::::::::::::::::::::::::::::::::
:: first parameter is a path to the graph xml
set graphXmlPath=“D:\IMAGES\C2RCC\c2rcc_olci.xml”
:: second parameter is a path to a parameter file
:: set parameterFilePath=“D:\IMAGES\C2RCC\resample_20m.properties” I do not want to change any parameter now. Should I just skip this line?
:: use third parameter for path to source products
set sourceDirectory=D:\IMAGES\SEN_L1\BC_coast\test
:: if sourceDirectory ends with ‘’ remove it
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%
:: use third parameter for path to target products
set targetDirectory=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch
:: if targetDirectory ends with ‘’ remove it
if %targetDirectory:~-1%==\ set targetDirectory=%targetDirectory:~0,-1%
:: the fourth parameter is a file prefix for the target product name,
:: typically indicating the type of processing
set targetFilePrefix=C2RCC
:: Create the target directory
md D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch\test
::::::::::::::::::::::::::::::::::::::::::::
:: Main processing
::::::::::::::::::::::::::::::::::::::::::::
:: double ‘%’ in batch file and only a single ‘%’ on command line
:: ‘/D’ is for directories like Sentinel data. Remove ‘/D’ when you open files.
for /D /R %sourceDirectory% %%F in (S3*.SEN3/xfdumanifest.xml) do (
echo.
:: ‘~fF’ means abolute path of ‘F’
set sourceFile=%%~fF
echo Processing !sourceFile!
:: ‘~nF’ means filename without extension of ‘F’
set targetFile=%targetDirectory%%targetFilePrefix%_%%~nF.dim
set procCmd=%gptPath% %graphXmlPath% -e -p %parameterFilePath% -t “!targetFile!” “!sourceFile!”
call !procCmd!
)
What do I still need to edit to make it work? When I write in the command line prompt: D:\IMAGES\C2RCC\processDataset.bat c2rcc_olci.xml D:\IMAGES\SEN_L1\BC_coast\test D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch C2RCC This is what I get and nothing happens:
setlocal ENABLEDELAYEDEXPANSION
set gptPath=“C:\Program Files\snap\bin\gpt.exe”
set graphXmlPath=\IMAGES\C2RCC\c2rcc_olci.xml
set sourceDirectory=D:\IMAGES\SEN_L1\BC_coast\test
if t == \ set sourceDirectory=D:\IMAGES\SEN_L1\BC_coast\tes
set targetDirectory=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch
if h == \ set targetDirectory=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batc
set targetFilePrefix=C2RCC
md D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch\test
for / %F in (S3*.SEN3/xfdumanifest.xml) do (
echo.
set sourceFile=%~fF
echo Processing !sourceFile!
set targetFile=D:\IMAGES\SEN_L1\BC_coast\c2rcc_batch\C2RCC_%~nF.dim
set procCmd=“C:\Program Files\snap\bin\gpt.exe” \IMAGES\C2RCC\c2rcc_olci.xml -e -p -t “!targetFile!” “!sourceFile!”
call !procCmd!
)
I would appreciate any help on how to start this kind of work.
I am having the exact same issue. Did you manage to find solution to this? If yes, I would appreciate it much if you could share with me on how?
Below is how I configured my batch processing file:
@echo off
rem enable delayed expansion - used to resolve variable in loop
rem variable has to be used with ‘!’ instead of ‘%’
setlocal ENABLEDELAYEDEXPANSION
::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::
rem adapt this path to your needs
set gptPath=“c:\Program Files\snap\bin\gpt.exe”
::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling
::::::::::::::::::::::::::::::::::::::::::::
rem first parameter is a path to the graph xml
set graphXmlPath=C:\Users\au\Desktop\Trial\Reprojection_SU.xml
rem second parameter is a path to a parameter file
set parameterFilePath=""
rem use third parameter for path to source products
set sourceDirectory=“C:\Users\au\Desktop\Trial\Slave”
set template=“C:\Users\au\Desktop\Trial\reprojected_S3A_OL_2_WFR____20171231T091907_20171231T092031_20180101T163947_0084_026_150_2075_MAR_O_NT_002.nc”
rem use third parameter for path to target products
set targetDirectory=C:\Users\au\Desktop\Trial\Reprojected
rem the fourth parameter is a file prefix for the target product name,
rem typically indicating the type of processing
set targetFilePrefix=reproj_
rem Create the target directory
rem md targetDirectory
::::::::::::::::::::::::::::::::::::::::::::
:: Main processing
::::::::::::::::::::::::::::::::::::::::::::
rem double ‘' in batch file and only a single '’ on command line
for / %F in (S3*.SEN3/xfdumanifest.xml) do (
rem ‘~fF’ means abolute path of ‘F’
set sourceFile=%%~fF
rem ‘~nF’ means filename without extension of ‘F’
set targetFile=%targetDirectory%%targetFilePrefix%_%%~nF.nc
set procCmd=gptPathgraphXmlPath -e -f NetCDF4-BEAM -t “!targetFile!” -Ssource="!sourceFile!" -Smaster="!template!"
call !procCmd!
)