OLCI C2RCC batch failure

yonisherman · February 21, 2023, 8:40pm

Hi everyone,
I am having trouble running C2RCC on OLCI. I have set up a processDataset.bat file and a c2rcc_olci_proc xml file (see attached).
when I run the commend

.\processDataset.bat .\c2rcc_olci_proc.xml "H:\BOL_satellite_data\Arctic\YK_S3\level_1\test\" "H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017\" C2RCC

it loops over the files I put in a test folder (2 files) but for each file I get an Error: -t (The system cannot find the file specified).

I am not sure why this is happing as the file paths seem to be correct. I tired pointing to the S3 folder itself as well as the xfdumanifest.xml file but get the same error. I tried looking through the forum but couldn’t find a similar issue.

Thank you for your help!

here is the full output from the run:

(satenv) PS C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP> .\processDataset.bat .\c2rcc_olci_proc.xml "H:\BOL_satellite_data\Arctic\YK_S3\level_1\test\" "H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017\" C2RCC

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>setlocal ENABLEDELAYEDEXPANSION

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>set gptPath="C:\Users\bol_remote_sensing\AppData\Local\Programs\snap\bin\gpt.exe"

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>set graphXmlPath=.\c2rcc_olci_proc.xml

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>set sourceDirectory=H:\BOL_satellite_data\Arctic\YK_S3\level_1\test\

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>if \ == \ set sourceDirectory=H:\BOL_satellite_data\Arctic\YK_S3\level_1\test

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>set targetDirectory=H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017\

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>if \ == \ set targetDirectory=H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>set targetFileSuffix=C2RCC

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>md H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017
A subdirectory or file H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017 already exists.

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>for / %F in (S3*_x) do (
echo.
 set sourceFile=%~fF\xfdumanifest.xml
 echo Processing !sourceFile!
 set targetFile=H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017\%~nF_C2RCC.nc
 set procCmd="C:\Users\bol_remote_sensing\AppData\Local\Programs\snap\bin\gpt.exe" .\c2rcc_olci_proc.xml -e -p  -t "!targetFile!" "!sourceFile!" -f NetCDF4-CF
 call !procCmd!
)

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>(
echo.
 set sourceFile=H:\BOL_satellite_data\Arctic\YK_S3\level_1\test\S3A_OL_1_EFR____20171030T225128_20171030T225253_20180721T045623_0085_024_044_1800_LR2_R_NT_002_x\xfdumanifest.xml
 echo Processing !sourceFile!
 set targetFile=H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017\S3A_OL_1_EFR____20171030T225128_20171030T225253_20180721T045623_0085_024_044_1800_LR2_R_NT_002_x_C2RCC.nc
 set procCmd="C:\Users\bol_remote_sensing\AppData\Local\Programs\snap\bin\gpt.exe" .\c2rcc_olci_proc.xml -e -p  -t "!targetFile!" "!sourceFile!" -f NetCDF4-CF
 call !procCmd!
)

Processing H:\BOL_satellite_data\Arctic\YK_S3\level_1\test\S3A_OL_1_EFR____20171030T225128_20171030T225253_20180721T045623_0085_024_044_1800_LR2_R_NT_002_x\xfdumanifest.xml
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.2.1 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.2.1 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
java.io.FileNotFoundException: -t (The system cannot find the file specified)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at java.io.FileReader.<init>(FileReader.java:58)
        at org.esa.snap.core.gpf.main.DefaultCommandLineContext.createReader(DefaultCommandLineContext.java:101)
        at com.bc.ceres.metadata.MetadataResourceEngine.readResource(MetadataResourceEngine.java:94)
        at org.esa.snap.core.gpf.main.CommandLineTool.getRawParameterMap(CommandLineTool.java:498)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:334)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
        at org.esa.snap.core.gpf.main.GPT.run(GPT.java:59)
        at org.esa.snap.core.gpf.main.GPT.main(GPT.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
        at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
        at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
        at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
        at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:94)
        at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:25)

Error: -t (The system cannot find the file specified)

C:\Users\bol_remote_sensing\remote_sensing\sherman\code\SNAP>(
echo.
 set sourceFile=H:\BOL_satellite_data\Arctic\YK_S3\level_1\test\S3A_OL_1_EFR____20171031T204543_20171031T204843_20171102T014240_0180_024_057_1979_LN1_O_NT_002_x\xfdumanifest.xml
 echo Processing !sourceFile!
 set targetFile=H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017\S3A_OL_1_EFR____20171031T204543_20171031T204843_20171102T014240_0180_024_057_1979_LN1_O_NT_002_x_C2RCC.nc
 set procCmd="C:\Users\bol_remote_sensing\AppData\Local\Programs\snap\bin\gpt.exe" .\c2rcc_olci_proc.xml -e -p  -t "!targetFile!" "!sourceFile!" -f NetCDF4-CF
 call !procCmd!
)

Processing H:\BOL_satellite_data\Arctic\YK_S3\level_1\test\S3A_OL_1_EFR____20171031T204543_20171031T204843_20171102T014240_0180_024_057_1979_LN1_O_NT_002_x\xfdumanifest.xml
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.2.1 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.2.1 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
java.io.FileNotFoundException: -t (The system cannot find the file specified)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at java.io.FileReader.<init>(FileReader.java:58)
        at org.esa.snap.core.gpf.main.DefaultCommandLineContext.createReader(DefaultCommandLineContext.java:101)
        at com.bc.ceres.metadata.MetadataResourceEngine.readResource(MetadataResourceEngine.java:94)
        at org.esa.snap.core.gpf.main.CommandLineTool.getRawParameterMap(CommandLineTool.java:498)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:334)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
        at org.esa.snap.core.gpf.main.GPT.run(GPT.java:59)
        at org.esa.snap.core.gpf.main.GPT.main(GPT.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
        at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
        at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
        at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
        at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:94)
        at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:25)

Error: -t (The system cannot find the file specified)

c2rcc__olci_proc.xml (1.6 KB)
processDataset.bat (1.8 KB)

marpet · February 22, 2023, 9:08am

Hi Jonathan,

have you tried it without the quotes around the paths on the command line?
Just like this:
.\processDataset.bat .\c2rcc_olci_proc.xml H:\BOL_satellite_data\Arctic\YK_S3\level_1\test H:\BOL_satellite_data\Arctic\YK_S3\level_2\C2RCC\2017 C2RCC

yonisherman · February 22, 2023, 3:45pm

Thank you for the reply. that pointed me in the right direction, which led me to realize I was still calling a parameter file in the processDataset file. after removing that rt of the procCmd line (‘-p’) I ran into some HDF5 error that was output when the C2RCC was working. I found a similar issue in another thread mentioning version issues with anaconda. when I ran the gpt from the base environment rather than my virtual one it ran. I still get a comment in the middle of each file run after reaching some percentage:
Processing H:\BOL_satellite_data\Arctic\YK_S3\level_1\2017\S3A_OL_1_EFR____20170403T215347_20170403T215647_20180413T222853_0179_016_129_1800_LR2_R_NT_002_x\xfdumanifest.xml INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.6.2 found on system. Internal GDAL 3.2.1 from distribution will be used. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.2.1 set to be used by SNAP. INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Executing processing graph INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.2.1 set to be used by SNAP. INFO: org.esa.s3tbx.c2rcc.olci.C2rccOlciOperator: c2rcc initial tile : null, configured tile: java.awt.Dimension[width=1217,height=1023] INFO: org.hsqldb.persist.Logger: dataFileCache open start 12%.25%....35%.7248 [main] INFO serverStartup - Nc4Iosp: NetCDF-4 C library loaded (jna_path='C:\Users\bol_remote_sensing\.snap\auxdata\netcdf_natives\9.0.4\amd64', libname='netcdf'). 7264 [main] INFO serverStartup - NetcdfLoader: set log level: old=0 new=0 7264 [main] INFO serverStartup - Nc4Iosp: set log level: old=0 new=0 51%64%77%90%

the processing continues and files open fine in SNAP desktop so not sure what that is, but thought I’d raise it.

In any case thank you for pointing me in the right direction
Jonathan

marpet · February 22, 2023, 4:18pm

I am glad that I could help you.
The logs are done by the NetCDF library. Unfortunately, they cannot be disabled.
Your data should be OK.

Pang · April 5, 2023, 4:20pm

Hi marpet,
I use c2rcc processing Sentinel 3 OLCI data has been some problems, I intend to change a method. I want to retrieve chlorophyll a in clean lake water, in addition to c2rcc, what is the atmospheric correction method?

Pang · April 6, 2023, 6:24am

Hi marpet,
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT
In this link what is the “%” mean?

Should I add % into my path?

Any wrong steps of the screenshot that I sent to you above? How to use .bat correctly? What is the “more?” mean?

marpet · April 6, 2023, 7:10am

As said in the other thread. If you want to work with batch scripts, you should familiars yourself with its features and syntax. Links which help to learn batch scripting are given at the end of the page.

The % usually indicates a variable.

Here the third parameter given at the command line.
set sourceDirectory=%3

Here the variable sourceDirectory, including the string operations on it.
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%

Here the file variable F with file operation.
set sourceFile=%%~fF

Pang · April 21, 2023, 7:41am

Dear Marco,
Firstly thanks for your kind help. But there are some questions recently.
When .bat lake this,

there are some problems with “!procCmd!”

When I change to this,

there are some errors:

How to deal that?
Thanks a lot.

marpet · April 21, 2023, 8:08am

Hi,

The version with %procCmd% is not correct. It has to be !procCmd! within in a loop.
You can add before the line call !procCmd! an echo command, like
echo !procCmd!
This should show you how the variable is evaluated. And maybe you can see a mistake there.
The call can be disabled by add two colons add the beginning of the line.
:: call !procCmd!

Btw., what’s the meaning of
?

Pang · April 21, 2023, 8:37am

Dear Marco,
When there is echo !procCmd!

When there is call !procCmd!

The first command is
E:\processDataset_test.bat I:\Sentinel3\myGraph.xml I:\Sentinel3\c2rcc.properties I:\Sentinel3\input I:\Sentinel3\output test

marpet · April 21, 2023, 9:35am

Without knowing your myGraph.xml file you seem to have a Read node, which expects an $input parameter. You need to provide this with the command line call.
Change to the following:
-Pinput="!sourceFile"

Pang · April 21, 2023, 10:06am

Dear Marco,
Thanks!
Actually, I found there are some errors in my .XML.
Now I can run it!
But I’m confused for xml. My .xml is from SNAP, which I clipped a small rectangle. Because I only need that one small area, and here’s what I do:
I bulit a subset

and wirte it for NetCDF4-CF.

Is that right?
But why there is only one image to output?

Thanks again!

marpet · April 21, 2023, 12:14pm

It could be that there is a fixed output coded in the xml and the target parameter is not considered at all.

Instead of -t %targetFile% you should use -Poutput=%targetFile%
And in the write node of the xml you can use ${output} for the file parameter.