How to Process multiple files using C2RCC

Hello,

I want to process multi files using C2RCC algorithm. Here is the graph xml file.

“\Testing_multi_files\gpt_olcic2rcc_graphsxml.xml” -t \Testing_multi_files\temp\test.nc -f “NetCDF4-CF” Testing_multi_files\S3A_OL_1_EFR____20180826T062228_20180826T062528_20180827T104339_0179_035_077_1800_MAR_O_NT_001.SEN3\xfdumanifest.xml

<graph id="trygraph">
  <version>1.0</version>
  <node id="trynode">
    <operator>c2rcc.olci</operator>
    <sources>
	        <sourceProduct>${sourceProduct}</sourceProduct>
<!--       
       <tomsomiStartProduct>${tomsomiStartProduct}</tomsomiStartProduct>
       <tomsomiEndProduct>${tomsomiEndProduct}</tomsomiEndProduct>
       <ncepStartProduct>${ncepStartProduct}</ncepStartProduct>
       <ncepEndProduct>${ncepEndProduct}</ncepEndProduct>
-->
    </sources>
     <parameters>
      
      <salinity>35</salinity>
      <temperature>15</temperature>
      <ozone>330</ozone>
      <press>1000</press>
      <TSMfakBpart>1.72</TSMfakBpart>
      <TSMfakBwit>3.1</TSMfakBwit>
      <CHLexp>1.04</CHLexp>
      <CHLfak>21</CHLfak>
      <thresholdRtosaOOS>0.005</thresholdRtosaOOS>
      <thresholdAcReflecOos>0.1</thresholdAcReflecOos>
      <thresholdCloudTDown865>0.955</thresholdCloudTDown865>
      <outputAsRrs>false</outputAsRrs>
      <deriveRwFromPathAndTransmittance>false</deriveRwFromPathAndTransmittance>
      <useEcmwfAuxData>true</useEcmwfAuxData>
      <outputRtoa>false</outputRtoa>
      <outputRtosaGc>false</outputRtosaGc>
      <outputRtosaGcAann>false</outputRtosaGcAann>
      <outputRpath>false</outputRpath>
      <outputTdown>false</outputTdown>
      <outputTup>false</outputTup>
      <outputAcReflectance>true</outputAcReflectance>
      <outputRhown>false</outputRhown>
      <outputOos>false</outputOos>
      <outputKd>false</outputKd>
      <outputUncertainties>true</outputUncertainties>
    </parameters>
  </node>
</graph>

The following wiki page explains how to do bulk processingwith gpt
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

Thaks Marpet. It creates a file (resampled 20) but its empty.

C:\Users\mas108\Downloads\aaa\CA\Sentinel2>processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>setlocal ENABLEDELAYEDEXPANSION

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set gptPath=“C:\Program Files\snap\bin\gpt.exe”

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set graphXmlPath=resample_s2.xml

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set parameterFilePath=resample_20m.properties

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set sourceDirectory=\input

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if t == \ set sourceDirectory=\inpu

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetDirectory=resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if m == \ set targetDirectory=resampled20

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetFilePrefix=

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>md resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>for / F in (S2*.SAFE) do ( echo. set sourceFile=~fF
echo Processing !sourceFile!
set targetFile=resampled20m_%~nF.dim
set procCmd=“C:\Program Files\snap\bin\gpt.exe” resample_s2.xml -e -p resample_20m.properties -t “!targetFile!” “!sourceFile!”
call !procCmd!
)

What would you like to process, OLCI or MSI?

I would like to process MSI.

gpt processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m
Error message:
No product reader found for ‘C:\Users\mas108\Downloads\aaa\CA\Sentinel2\resample_s2.xml’

Please, read the link marpet sent about batch processing with GPT and make a more concrete question about the xml or the command line order.

I have read the link sent by Marco. I want to process multi MS1 scences in one go. I have exactly folowed the tutorial sent by Marco.

Ok, first you have to know that the graph you are using is not the C2RCC but the Resample. Of course to apply C2RCC to MSI you have to resample first (it could be 20m as you wrote). You will then have to use the resampled product as input in the C2RCC graph. You could do it in one go using a single xml file. The trick is to do the process first in SNAP and then copy the parameters (or save them) into the xml file.
Some threads you can use:


After your xml can be applied to one image using the command line, you can try the .bat file to repeat the process on several images.

Hi,

Running this command

gpt processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m

I got this error:
No product reader found for ‘C:\Users\mas108\Downloads\aaa\CA\Sentinel2\resample_s2.xml’

I want to process MSI (multi files in one go) using C2RCC but first I need to resample them. Kindly advise.

gpt processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m
This call is not correct. You need to remove the ‘gpt’ at the beginning. gpt is called by the batch file.
You can also do both, resampling and C2RCC processing, in one graph.
How to create graphs is explained here: https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503590/Creating+a+GPF+Graph

By running the command without gpt, it creates a new folder resampled20 but the algorithm doesnt run. The link you have shared is for running C2RCC but I am in step-1 (resampling data first). KIndly advise.

processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m

Output:
(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>setlocal ENABLEDELAYEDEXPANSION

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set gptPath=“C:\Users\mas108\snap\bin\gpt.exe”

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set graphXmlPath=resample_s2.xml

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set parameterFilePath=resample_20m.properties

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set sourceDirectory=\input

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if t == \ set sourceDirectory=\inpu

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetDirectory=resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if m == \ set targetDirectory=resampled20

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetFilePrefix=

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>md resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>for / F in (S2*.SAFE) do ( echo. set sourceFile=~fF
echo Processing !sourceFile!
set targetFile=resampled20m_%~nF.dim
set procCmd=“C:\Users\mas108\snap\bin\gpt.exe” resample_s2.xml -e -p resample_20m.properties -t “!targetFile!” “!sourceFile!”
call !procCmd!
)

Maybe you try to use absolute paths for the input and output directories.
I guess it can’t find data products in ‘\input’. So try an absolute path.

Dear everyone,

1.when i try to S2Resampling , it works, but when i try c2rcc, it said it could not find the xml file. thanks

./gpt S2Resampling -SsourceProduct=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE’ -Pdownsampling=‘Mean’ -PflagDownsampling=‘FlagMedianAnd’ -Presolution=‘20’ -Pupsampling=‘Bilinear’

2.when I try batch resample for sentinel2 , after that i will try C2RCC. but it fails at the beginning.
the details see error code.

error code:
[bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash
[bjiang@viz3]~/snap/bin%

I do not see in your command line that you are indicating which one is your target product, only your source. I do not see the. xml either. I mean, the gpt command line should be: ./gpt dir/_to your_S2Resampling.xml dir/_to your_source_product dir/_to your_target_product. The S2Resampling.xml will content of the resampling parameters in the xml file.

Your bash file order does not make sense. If you have all well written in your bash, you only need to run it, you do not need to specify anything else.

Dear abruescas,

what you mean that processDataset.bash? i just follow the webiste https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

it just not work. Is that make sense. thanks

Best

Binbin

Dear marpet,

when i try process resample for sentinel2, it not works, details in below. thanks

Best

Binbin

Case1 error
(binbin2) [bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

Case2 error
(binbin2) [bjiang@viz3]~/snap/bin% ./gpt processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH does not contain the current folder ‘.’. Its value is ‘/opt/modules/centos7/openslide/3.4.1/lib:/opt/modules/centos7/gcc/7.3.0/lib:/opt/modules/centos7/gcc/7.3.0/lib64:/opt/modules/centos7/libs:/opt/modules/centos7/libX’.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
WARNING: org.esa.snap.core.util.EngineVersionCheckActivator: A new SNAP version is available for download.
Currently installed 7.0, available is 8.0.0.
Please visit http://step.esa.int

org.esa.snap.core.gpf.OperatorException: No product reader found for ‘/nfs3-home/bjiang/snap/bin/resample_s2.xml’
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.readProduct(DefaultCommandLineContext.java:57)
at org.esa.snap.core.gpf.main.CommandLineTool.readProduct(CommandLineTool.java:521)
at org.esa.snap.core.gpf.main.CommandLineTool.addProduct(CommandLineTool.java:466)
at org.esa.snap.core.gpf.main.CommandLineTool.getSourceProductMap(CommandLineTool.java:454)
at org.esa.snap.core.gpf.main.CommandLineTool.runOperator(CommandLineTool.java:295)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:284)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

Error: No product reader found for ‘/nfs3-home/bjiang/snap/bin/resample_s2.xml’

Where have you stored the processDataset.bash?

In your first case, the error

zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

tells you that the command (the file is not found). Your working directory seems to be /snap/bin.
You can save the file there. or you change the current working directory to the folder where the bash file is already located. Then the command should work.

In your second case you call gpt. But the parameters do not match the expected values.

Dear marpet,

in first case, i put the file “processDataset.bash” , "resample_s2.xml " and “resample_20m.properties” stored in “snap/bin” thanks

Best

Binbin

Maybe the file is not marked as executable.

How to Make Bash Script Executable Using Chmod (linoxide.com)
Have a look at step 4.