How to Process multiple files using C2RCC

snap
#12

By running the command without gpt, it creates a new folder resampled20 but the algorithm doesnt run. The link you have shared is for running C2RCC but I am in step-1 (resampling data first). KIndly advise.

processDataset.bat resample_s2.xml resample_20m.properties \input \output resampled20m

Output:
(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>setlocal ENABLEDELAYEDEXPANSION

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set gptPath=“C:\Users\mas108\snap\bin\gpt.exe”

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set graphXmlPath=resample_s2.xml

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set parameterFilePath=resample_20m.properties

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set sourceDirectory=\input

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if t == \ set sourceDirectory=\inpu

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetDirectory=resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>if m == \ set targetDirectory=resampled20

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>set targetFilePrefix=

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>md resampled20m

(base) C:\Users\mas108\Downloads\aaa\CA\Sentinel2>for / F in (S2*.SAFE) do ( echo. set sourceFile=~fF
echo Processing !sourceFile!
set targetFile=resampled20m_%~nF.dim
set procCmd=“C:\Users\mas108\snap\bin\gpt.exe” resample_s2.xml -e -p resample_20m.properties -t “!targetFile!” “!sourceFile!”
call !procCmd!
)

#13

Maybe you try to use absolute paths for the input and output directories.
I guess it can’t find data products in ‘\input’. So try an absolute path.

#14

Dear everyone,

1.when i try to S2Resampling , it works, but when i try c2rcc, it said it could not find the xml file. thanks

./gpt S2Resampling -SsourceProduct=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE’ -Pdownsampling=‘Mean’ -PflagDownsampling=‘FlagMedianAnd’ -Presolution=‘20’ -Pupsampling=‘Bilinear’

2.when I try batch resample for sentinel2 , after that i will try C2RCC. but it fails at the beginning.
the details see error code.

error code:
[bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash
[bjiang@viz3]~/snap/bin%

C2RCC problem for MSI granules processing
#15

I do not see in your command line that you are indicating which one is your target product, only your source. I do not see the. xml either. I mean, the gpt command line should be: ./gpt dir/_to your_S2Resampling.xml dir/_to your_source_product dir/_to your_target_product. The S2Resampling.xml will content of the resampling parameters in the xml file.

Your bash file order does not make sense. If you have all well written in your bash, you only need to run it, you do not need to specify anything else.

#17

Dear abruescas,

what you mean that processDataset.bash? i just follow the webiste https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

it just not work. Is that make sense. thanks

Best

Binbin

#18

Dear marpet,

when i try process resample for sentinel2, it not works, details in below. thanks

Best

Binbin

Case1 error
(binbin2) [bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

Case2 error
(binbin2) [bjiang@viz3]~/snap/bin% ./gpt processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH does not contain the current folder ‘.’. Its value is ‘/opt/modules/centos7/openslide/3.4.1/lib:/opt/modules/centos7/gcc/7.3.0/lib:/opt/modules/centos7/gcc/7.3.0/lib64:/opt/modules/centos7/libs:/opt/modules/centos7/libX’.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
WARNING: org.esa.snap.core.util.EngineVersionCheckActivator: A new SNAP version is available for download.
Currently installed 7.0, available is 8.0.0.
Please visit http://step.esa.int

org.esa.snap.core.gpf.OperatorException: No product reader found for ‘/nfs3-home/bjiang/snap/bin/resample_s2.xml’
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.readProduct(DefaultCommandLineContext.java:57)
at org.esa.snap.core.gpf.main.CommandLineTool.readProduct(CommandLineTool.java:521)
at org.esa.snap.core.gpf.main.CommandLineTool.addProduct(CommandLineTool.java:466)
at org.esa.snap.core.gpf.main.CommandLineTool.getSourceProductMap(CommandLineTool.java:454)
at org.esa.snap.core.gpf.main.CommandLineTool.runOperator(CommandLineTool.java:295)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:284)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

Error: No product reader found for ‘/nfs3-home/bjiang/snap/bin/resample_s2.xml’

#19

Where have you stored the processDataset.bash?

In your first case, the error

zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

tells you that the command (the file is not found). Your working directory seems to be /snap/bin.
You can save the file there. or you change the current working directory to the folder where the bash file is already located. Then the command should work.

In your second case you call gpt. But the parameters do not match the expected values.

#20

Dear marpet,

in first case, i put the file “processDataset.bash” , "resample_s2.xml " and “resample_20m.properties” stored in “snap/bin” thanks

Best

Binbin

#21

Maybe the file is not marked as executable.

How to Make Bash Script Executable Using Chmod (linoxide.com)
Have a look at step 4.

#22

try to follow your suggestion.

(binbin2) [bjiang@viz3]~/snap/bin% chmod u+x processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
chmod: cannot access ‘resampled20m’: No such file or directory
(binbin2) [bjiang@viz3]~/snap/bin% chmod u+x processDataset.bash
(binbin2) [bjiang@viz3]~/snap/bin% processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
zsh: processDataset.bash: command not found…
zsh: command not found: processDataset.bash

#23

by the way, i fix the “processDataset.bash” for my gpt path, like below,

adapt this path to your needs

export PATH=~/home/bjiang/snap/bin:$PATH
gptPath=“gpt”

is that right? when i try it again, the error like this below
(binbin2) [bjiang@viz3]~/snap/bin% ./processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/orgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE/C2RCC” resampled20m
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
./processDataset.bash: line 53: gpt: command not found
(binbin2) [bjiang@viz3]~/snap/bin%

Best

Binbin

#24

It looks like works. thank you all. any suggestion for c2rcc atmospheric correction?

Best

Binbin

(binbin2) [bjiang@viz3]~/snap/bin% ./processDataset.bash resample_s2.xml resample_20m.properties “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/trytryorgin” “/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry” resampled20m
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Native library load failed.
java.lang.UnsatisfiedLinkError: /nfs3-home/bjiang/.snap/auxdata/gdal/gdal-3-0-X-jni/libgdalalljni.so: libgdal.so.26: cannot open shared object file: No such file or directory
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32619
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
…10%…20%…30%…40%…50%…60%…70%…80%…90% done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Native library load failed.
java.lang.UnsatisfiedLinkError: /nfs3-home/bjiang/.snap/auxdata/gdal/gdal-3-0-X-jni/libgdalalljni.so: libgdal.so.26: cannot open shared object file: No such file or directory
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32619
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
…10%…20%…30%…40%…50%…60%…70%…80%…90% done.
(binbin2) [bjiang@viz3]~/snap/bin%

#25

Good that it now works.

But the chmod command should actually look like:

chmod u+x processDataset.bash

Not more. Afterwards it can be invoked the normal way.

#26

Dear marpet,

after i get the sentinel2 20m , i want to c2rcc.msi process, but it not works. Could you have some suggestions? thanks

Best

Binbin
command line:
./gpt c2rcc.msi -SsourceProduct=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’ -PoutputRpath=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry/C2RCC’
error
(binbin2) [bjiang@viz3]~/snap/bin% ./gpt c2rcc.msi -SsourceProduct=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’ -PoutputRpath=’/home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry/C2RCC’
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
Native library load failed.
java.lang.UnsatisfiedLinkError: /nfs3-home/bjiang/.snap/auxdata/gdal/gdal-3-0-X-jni/libgdalalljni.so: libgdal.so.26: cannot open shared object file: No such file or directory
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
org.esa.snap.core.gpf.OperatorException: No product reader found for ‘/nfs3-home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.readProduct(DefaultCommandLineContext.java:57)
at org.esa.snap.core.gpf.main.CommandLineTool.readProduct(CommandLineTool.java:534)
at org.esa.snap.core.gpf.main.CommandLineTool.addProduct(CommandLineTool.java:479)
at org.esa.snap.core.gpf.main.CommandLineTool.getSourceProductMap(CommandLineTool.java:467)
at org.esa.snap.core.gpf.main.CommandLineTool.runOperator(CommandLineTool.java:295)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:284)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)

Error: No product reader found for ‘/nfs3-home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry’
(binbin2) [bjiang@viz3]~/snap/bin%

#27

The error explains it.
The path you have specified for the source product does not point to a product. But only to a directory, I guess.

#28

Dear marpet,

how could fix the problem. thanks. i have no idea about it. the “C2RCCtry” directory already has the after bash resample 20m sentinel2 data “resampled20m_S2A_MSIL1C_20160106T153632.data” and “resampled20m_S2A_MSIL1C_20160106T153632.dim” . it should works with ./gpt c2rcc.msi .

Thanks.

Binbin

#29

Have you modified the processDataset.bash file? You could compare it with the original file and maybe you see a difference.

#30

Dear marpet,

thanks for your reply fast. How could i modified the processDataset.bash? could you provide an example? thanks a lot.

Best

Binbin

#31

Sorry, but I can’t provide more help which exceeds the original guide
Bulk Processing with GPT - SNAP
Maybe you start with the basics of scripting on unix. This might help.
Shell Scripting Tutorial: How to Create Shell Script in Linux/Unix (guru99.com)

#32

A post was split to a new topic: How to get CDOM (Coloured Dissolved Organic Material) and DOC (Dissolved Organic Carbon)