How to Process multiple files using C2RCC

abruescas · August 21, 2019, 3:25pm

Ok, first you have to know that the graph you are using is not the C2RCC but the Resample. Of course to apply C2RCC to MSI you have to resample first (it could be 20m as you wrote). You will then have to use the resampled product as input in the C2RCC graph. You could do it in one go using a single xml file. The trick is to do the process first in SNAP and then copy the parameters (or save them) into the xml file.
Some threads you can use:

After your xml can be applied to one image using the command line, you can try the .bat file to repeat the process on several images.

Asim12 · August 22, 2019, 7:22am

Hi,

Running this command

gpt processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m

I got this error:
No product reader found for ‘C:\Users\mas108\Downloads\aaa\CA\Sentinel2\resample_s2.xml’

I want to process MSI (multi files in one go) using C2RCC but first I need to resample them. Kindly advise.

marpet · August 22, 2019, 7:46am

gpt processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m
This call is not correct. You need to remove the ‘gpt’ at the beginning. gpt is called by the batch file.
You can also do both, resampling and C2RCC processing, in one graph.
How to create graphs is explained here: https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503590/Creating+a+GPF+Graph

Asim12 · August 22, 2019, 8:16am

By running the command without gpt, it creates a new folder resampled20 but the algorithm doesnt run. The link you have shared is for running C2RCC but I am in step-1 (resampling data first). KIndly advise.

processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m

Output:
(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>setlocal ENABLEDELAYEDEXPANSION

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set gptPath=“C:\Users\mas108\snap\bin\gpt.exe”

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set graphXmlPath=resample_s2.xml

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set parameterFilePath=resample_20m.properties

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set sourceDirectory=\input

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if t == \ set sourceDirectory=\inpu

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetDirectory=resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if m == \ set targetDirectory=resampled20

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetFilePrefix=

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>md resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>for / F in (S2*.SAFE) do ( echo. set sourceFile=~fF
echo Processing !sourceFile!
set targetFile=resampled20m_%~nF.dim
set procCmd=“C:\Users\mas108\snap\bin\gpt.exe” resample_s2.xml -e -p resample_20m.properties -t “!targetFile!” “!sourceFile!”
call !procCmd!
)

marpet · August 22, 2019, 8:59am

Maybe you try to use absolute paths for the input and output directories.
I guess it can’t find data products in ‘\input’. So try an absolute path.

binbinjiang · February 27, 2021, 8:59am

Dear everyone,

1.when i try to S2Resampling , it works, but when i try c2rcc, it said it could not find the xml file. thanks

./gpt S2Resampling -SsourceProduct=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE’ -Pdownsampling=‘Mean’ -PflagDownsampling=‘FlagMedianAnd’ -Presolution=‘20’ -Pupsampling=‘Bilinear’

2.when I try batch resample for sentinel2 , after that i will try C2RCC. but it fails at the beginning.
the details see error code.

error code:
[bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash
[bjiang@viz3]~/snap/bin%

abruescas · February 27, 2021, 9:59am

I do not see in your command line that you are indicating which one is your target product, only your source. I do not see the. xml either. I mean, the gpt command line should be: ./gpt dir/_to your_S2Resampling.xml dir/_to your_source_product dir/_to your_target_product. The S2Resampling.xml will content of the resampling parameters in the xml file.

Your bash file order does not make sense. If you have all well written in your bash, you only need to run it, you do not need to specify anything else.

binbinjiang · March 2, 2021, 7:59pm

Dear abruescas,

what you mean that processDataset.bash? i just follow the webiste https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

it just not work. Is that make sense. thanks

Best

Binbin

binbinjiang · March 2, 2021, 9:13pm

Dear marpet,

when i try process resample for sentinel2, it not works, details in below. thanks

Best

Binbin

Case1 error
(binbin2) [bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

Case2 error
(binbin2) [bjiang@viz3]~/snap/bin% ./gpt processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH does not contain the current folder ‘.’. Its value is ‘/opt/modules/centos7/openslide/3.4.1/lib:/opt/modules/centos7/gcc/7.3.0/lib:/opt/modules/centos7/gcc/7.3.0/lib64:/opt/modules/centos7/libs:/opt/modules/centos7/libX’.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
WARNING: org.esa.snap.core.util.EngineVersionCheckActivator: A new SNAP version is available for download.
Currently installed 7.0, available is 8.0.0.
Please visit http://step.esa.int

org.esa.snap.core.gpf.OperatorException: No product reader found for ‘/nfs3-home/bjiang/snap/bin/resample_s2.xml’
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.readProduct(DefaultCommandLineContext.java:57)
at org.esa.snap.core.gpf.main.CommandLineTool.readProduct(CommandLineTool.java:521)
at org.esa.snap.core.gpf.main.CommandLineTool.addProduct(CommandLineTool.java:466)
at org.esa.snap.core.gpf.main.CommandLineTool.getSourceProductMap(CommandLineTool.java:454)
at org.esa.snap.core.gpf.main.CommandLineTool.runOperator(CommandLineTool.java:295)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:284)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

Error: No product reader found for ‘/nfs3-home/bjiang/snap/bin/resample_s2.xml’

marpet · March 3, 2021, 9:47am

Where have you stored the processDataset.bash?

In your first case, the error

zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

tells you that the command (the file is not found). Your working directory seems to be /snap/bin.
You can save the file there. or you change the current working directory to the folder where the bash file is already located. Then the command should work.

In your second case you call gpt. But the parameters do not match the expected values.

binbinjiang · March 3, 2021, 2:19pm

Dear marpet,

in first case, i put the file “processDataset.bash” , "resample_s2.xml " and “resample_20m.properties” stored in “snap/bin” thanks

Best

Binbin

marpet · March 3, 2021, 3:13pm

Maybe the file is not marked as executable.

How to Make Bash Script Executable Using Chmod (linoxide.com)
Have a look at step 4.

binbinjiang · March 3, 2021, 3:52pm

try to follow your suggestion.

(binbin2) [bjiang@viz3]~/snap/bin% chmod u+x processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
chmod: cannot access ‘resampled20m’: No such file or directory
(binbin2) [bjiang@viz3]~/snap/bin% chmod u+x processDataset.bash
(binbin2) [bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

binbinjiang · March 3, 2021, 4:08pm

by the way, i fix the “processDataset.bash” for my gpt path, like below,

adapt this path to your needs

export PATH=~/home/bjiang/snap/bin:$PATH
gptPath=“gpt”

is that right? when i try it again, the error like this below
(binbin2) [bjiang@viz3]~/snap/bin% ./processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
(binbin2) [bjiang@viz3]~/snap/bin%

Best

Binbin

binbinjiang · March 3, 2021, 8:41pm

It looks like works. thank you all. any suggestion for c2rcc atmospheric correction?

Best

Binbin

(binbin2) [bjiang@viz3]~/snap/bin% ./processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/trytryorgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry” resampled20m
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Native library load failed.
java.lang.UnsatisfiedLinkError: /nfs3-home/bjiang/.snap/auxdata/gdal/gdal-3-0-X-jni/libgdalalljni.so: libgdal.so.26: cannot open shared object file: No such file or directory
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32619
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
…10%…20%…30%…40%…50%…60%…70%…80%…90% done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Native library load failed.
java.lang.UnsatisfiedLinkError: /nfs3-home/bjiang/.snap/auxdata/gdal/gdal-3-0-X-jni/libgdalalljni.so: libgdal.so.26: cannot open shared object file: No such file or directory
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32619
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
…10%…20%…30%…40%…50%…60%…70%…80%…90% done.
(binbin2) [bjiang@viz3]~/snap/bin%

marpet · March 4, 2021, 7:08am

Good that it now works.

But the chmod command should actually look like:

chmod u+x processDataset.bash

Not more. Afterwards it can be invoked the normal way.

binbinjiang · March 5, 2021, 4:17pm

Dear marpet,

after i get the sentinel2 20m , i want to c2rcc.msi process, but it not works. Could you have some suggestions? thanks

Best

Binbin
command line:
./gpt c2rcc.msi -SsourceProduct=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’ -PoutputRpath=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry/C2RCC’
error
(binbin2) [bjiang@viz3]~/snap/bin% ./gpt c2rcc.msi -SsourceProduct=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’ -PoutputRpath=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry/C2RCC’
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Native library load failed.
java.lang.UnsatisfiedLinkError: /nfs3-home/bjiang/.snap/auxdata/gdal/gdal-3-0-X-jni/libgdalalljni.so: libgdal.so.26: cannot open shared object file: No such file or directory
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
org.esa.snap.core.gpf.OperatorException: No product reader found for ‘/nfs3-home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.readProduct(DefaultCommandLineContext.java:57)
at org.esa.snap.core.gpf.main.CommandLineTool.readProduct(CommandLineTool.java:534)
at org.esa.snap.core.gpf.main.CommandLineTool.addProduct(CommandLineTool.java:479)
at org.esa.snap.core.gpf.main.CommandLineTool.getSourceProductMap(CommandLineTool.java:467)
at org.esa.snap.core.gpf.main.CommandLineTool.runOperator(CommandLineTool.java:295)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:284)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)

Error: No product reader found for ‘/nfs3-home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’
(binbin2) [bjiang@viz3]~/snap/bin%

marpet · March 7, 2021, 8:02am

The error explains it.
The path you have specified for the source product does not point to a product. But only to a directory, I guess.

binbinjiang · March 7, 2021, 7:52pm

Dear marpet,

how could fix the problem. thanks. i have no idea about it. the “C2RCCtry” directory already has the after bash resample 20m sentinel2 data “resampled20m_S2A_MSIL1C_20160106T153632.data” and “resampled20m_S2A_MSIL1C_20160106T153632.dim” . it should works with ./gpt c2rcc.msi .

Thanks.

Binbin

marpet · March 8, 2021, 7:33am

Have you modified the processDataset.bash file? You could compare it with the original file and maybe you see a difference.