How run sentinel2 by batch c2rcc

Dear market,

here is my xml. it works. has three questions

  1. but I want to aux data auto download by NASA like(etc O3,NO2). how could I do ?
  2. I just save the Rrs data for netCDF file in a file, not like rrs_B1.img,rrs_B2.img…
  3. I already have bacthprocess.bash now, how could I use the bash file to try run it for lots of image,

Thanks a lot

Best

Binbin

my xml:

<graph id="Graph">
  <version>1.0</version>
  <node id="S2Resampling">
    <operator>S2Resampling</operator>
    <sources>
      <sourceProduct>${sourceProduct}</sourceProduct>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <resolution>20</resolution>
      <upsampling>Bilinear</upsampling>
      <downsampling>First</downsampling>
      <flagDownsampling>First</flagDownsampling>
      <resampleOnPyramidLevels>true</resampleOnPyramidLevels>
    </parameters>
  </node>
  <node id="c2rcc_msi">
    <operator>c2rcc.msi</operator>
    <sources>
      <sourceProduct refid="S2Resampling"/>
    </sources>
    <parameters>
      <salinity>35</salinity>
      <temperature>18</temperature>
      <outputRtoa>false</outputRtoa>
      <outputRtosaGc>false</outputRtosaGc>
      <outputAsRrs>true</outputAsRrs>
    </parameters>
   </node>
</graph>

msi_res20_c2rcc_bin.xml (881 Bytes)
[bjiang@viz3]~/snap/bin% ./gpt msi_res20_c2rcc_bin.xml -t /home/bjiang/Sentinel2/getOC-master-3/2016woodram/C2RCCtry/C2RCC/c2rcc /home/bjiang/Sentinel2/getOC-master-3/2016woodram/trytryorgin/S2A_MSIL1C_20160106T153632_N0201_R111_T19TCJ_20160106T153739.SAFE
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32619
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
.INFO: org.esa.s3tbx.c2rcc.ancillary.AtmosphericAuxdataBuilder: Atmospheric auxdata product can’t be used. At least one is not specified. Using constant values for ozone (330.0) and surface pressure (1000.0).
…10%…20%…30%…40%…50%…60%…70%…80%…90% done.
[bjiang@viz3]~/snap/bin%

1) The automatic download of the NASA auxdata is currently not working. A user account is now required for downloading the data and this is currently not considered in the processor.
[SIIITBX-369] Auxillary data is not automatically downloaded anymore in C2RCC - JIRA (atlassian.net)
But you can still download them manually and place the files in a directory and specify this directory via the atmosphericAuxDataPath parameter.
2) Here I don’t understand the question. Do you want to rename the bands? If so, you can use the Merge operator. You can get help if you call git on the command line:


3) If you have the bash file, you have everything to you need to process lot of images.
It is all explained her: Bulk Processing with GPT - SNAP - SNAP Wiki (atlassian.net)

Dear marpet,

thank you for your reply. The second question is that I merge the all of band. save one file with the format netCDF file. Is that possible? thanks

by the way, I want to another SNAP vision.6. Is that possible the same software SNAP has two visions in one computer?

Best

Binbin

You can edit the bash file

At line 52-53 you will find this:

  targetFile="${targetDirectory}/${targetFilePrefix}_$(removeExtension "$(basename ${F})").dim"
  ${gptPath} ${graphXmlPath} -e -p ${parameterFilePath} -t ${targetFile} ${sourceFile}

Change them to this:

  targetFile="${targetDirectory}/${targetFilePrefix}_$(removeExtension "$(basename ${F})").nc"
  ${gptPath} ${graphXmlPath} -e -p ${parameterFilePath} -t ${targetFile} -f NetCDF4-CF ${sourceFile}

Having two different versions of SNAP installed is not recommend it will probably lead to errors.

The NASA downloads are now using an “EarthData SSO” (single sign-on) that redirects the initial login to the SSO login page and then back to the data URL. This results in many people getting an html login page instead of the data file they expected. Wget[2] has an option to change the extension to match the file type, or, using a macOS or linux shell, you can check the type of suspiciously small files using the file command.

NASA uses recent encryption methods so older software often fails on either the encryption methods or the multiple redirects.

The user account is free. When you set up the account you need to “register” for the data you will be using. See OC Download Methods. The old Ocean Color Forum has been replaced with an EarthData forum which
is the best place for help downloading NASA data.

If you are using linux or macOS, NASA’s OCSSW processing system has some tools to automate downloads of ancillary data. NASA SeaDAS 8 (currently a beta release) will install a new version of the OCSSW software from the SNAP-8 based GUI which allows to download ancillary files from the GUI.

Dear marpet,

it works very well. And Do you know how to calibration at the beginning of process? like gain with band1 multiple with a factor 0.99; band 2 multiple with 1.03. thanks

Could I auto download the ERA-data like wind data, air pressure?

Best

Binbin

Dear marpet,

I will run lots of image of sentinel2, so want remove iop_apig, iop_adet,iop_agelb … but save “rrs_B1,rrs_B2…conc_tsm, lon,lat”. thanks

Best

Binbin

ERA data can be downloaded too. But are not used by c2rcc.

You can add a BandMaths operator to your graph and merge the result with the original product.
This way you can apply factor to the bands.

The bands iop_apig, iop_adet,iop_agelb cannot be disabled. They are the core output of the processor. But you can add a subset operator your graph and include only those bands you need.

Dear marpet,

when I run more than 2 images, like 30 images, it sometime not work for a few.

like error:

There is insufficient memory for Java Runtime Environment to continue .
Native memory allocation(map) failed to map 5567938560bytes for committing reserved memory. the detail see log attach in below.

hs_err_pid3651.log (174.5 KB)

Dear marpet,

show an error example image.

Best

Binbin

Dear marpet,

when I try SNAP 7.0 to batch sentinel2 with processdata.bash on Mac . but it works well in linux with SNAP8.0 ago. So the processdata only work for SNAP8? Could you give a hand? thanks

By the way, may I change “NetCDF4-CF” to “NetCDF4-BEAM” in processData.bash? thanks

Best

Binbin

error:
Binbin@mMISCLab bin % ./processDataset.bash msi_res20_c2rcc_bin.xml resample_20m.properties /Users/Binbin/Documents/trytry /Users/Binbin/Documents/trtry/

./processDataset.bash: line 51: realpath: command not found

sed: illegal option – r

usage: sed script [-Ealn] [-i extension] [file …]

sed [-Ealn] [-i extension] [-e script] … [-f script_file] … [file …]

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters

INFO: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: No distribution folder found on Macintosh.

INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.

Executing processing graph

done.

org.esa.snap.core.gpf.graph.GraphException: Missing source ‘${sourceProduct}’ in node ‘S2Resampling’

at org.esa.snap.core.gpf.graph.GraphContext.initNodeDependencies(GraphContext.java:113)

at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:90)

at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:64)

at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)

at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)

at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)

at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)

at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)

at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)

at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)

at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)

at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)

at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)

at org.esa.snap.runtime.Launcher.run(Launcher.java:51)

at org.esa.snap.runtime.Launcher.main(Launcher.java:31)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)

at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

Error: Missing source ‘${sourceProduct}’ in node 'S2Resampling’msi_res20_c2rcc_bin.xml (1.1 KB) processDataset.bash (1.5 KB)resample_20m.properties (55 Bytes)

In your .bash file you missed to put the directories to the everything (xml, input, outputs…)

Thanks abruescas for your reply. how could I fix it, thanks a lot.

Best

Binbin

You can start by changing the “$1” for the directory where you have the msi_res20_c2rcc_bin.xml file.

first parameter is a path to the graph xml

graphXmlPath="$1"

And continue with the other “$2”, “$3”, etc., with the directories of the different inputs and outputs.

Dear abruescas,

thanks a lot. but it still not work.

Best

BinbinprocessDataset.bash (1.6 KB)

Binbin@mMISCLab bin % ./processDataset.bash

./processDataset.bash: line 51: realpath: command not found

sed: illegal option – r

usage: sed script [-Ealn] [-i extension] [file …]

sed [-Ealn] [-i extension] [-e script] … [-f script_file] … [file …]

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters

INFO: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: No distribution folder found on Macintosh.

INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.

java.io.FileNotFoundException: /Applications/snap/bin (Is a directory)

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.(FileInputStream.java:138)

at java.io.FileInputStream.(FileInputStream.java:93)

at java.io.FileReader.(FileReader.java:58)

at org.esa.snap.core.gpf.main.DefaultCommandLineContext.createReader(DefaultCommandLineContext.java:101)

at com.bc.ceres.metadata.MetadataResourceEngine.readResource(MetadataResourceEngine.java:94)

at org.esa.snap.core.gpf.main.CommandLineTool.getRawParameterMap(CommandLineTool.java:485)

at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:337)

at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)

at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)

at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)

at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)

at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)

at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)

at org.esa.snap.runtime.Launcher.run(Launcher.java:51)

at org.esa.snap.runtime.Launcher.main(Launcher.java:31)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)

at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

Error: /Applications/snap/bin (Is a directory)

Binbin@mMISCLab bin %

realpath has a complicated history, so it is not surprising it isn’t available with the rather old bash version provided in macOS. GNU coreutils provides a reapath executable. On macOS it may be called grealpath. Some old bash installations provided a function:

$ type -a realpath
realpath is a function                                                                                                                                                                        
realpath ()                                                                                                                                                                                   
{                                                                                                                                                                                             
    f=$@;                                                                                                                                                                                     
    if [ -d "$f" ]; then                                                                                                                                                                      
        base="";                                                                                                                                                                              
        dir="$f";                                                                                                                                                                             
    else                                                                                                                                                                                      
        base="/$(basename "$f")";                                                                                                                                                             
        dir=$(dirname "$f");                                                                                                                                                                  
    fi;                                                                                                                                                                                       
    dir=$(cd "$dir" && /bin/pwd);                                                                                                                                                             
    echo "$dir$base"                                                                                                                                                                          
}     

Bash versions 4.4 and later have a loadable extension:

BASH_LOADABLES_PATH=${BASH_LOADABLES_PATH:-/usr/local/lib/bash:/usr/lib/bash}
enable -f realpath realpath

The -r option is specific to GNU sed and enables extended regular expressons. On macOS you might need -E instead of -r.

It is possible to configure macOS to use GNU versions of common utilities, but this can cause problems for scripts that are intended for use on a basic macOS configuration.

There is another possibility for mac users and command line gpt. It is a bit more work, but batch processing works well in this case. Instead of using the .bash file you can write your gpt command line on a shell file, indicating where is the gpt directory, the xml file, the output and input directories. I wrote an example below (from another case that uses S3, you should adapt it):

./gpt /Users/Anita/Desktop/SNAP_Forum/gpt_manacapuru.xml -t /Users/Anita/Desktop/SNAP_Forum/Downloads/S3A_OL_2_WFR____20191104T135002_20191104T135302_20191124T134508_0179_051_124_3060_MAR_O_NT_002_subset.dim -Ssource=/Users/Anita/Desktop/SNAP_Forum/Downloads/S3A_OL_2_WFR____20191104T135002_20191104T135302_20191124T134508_0179_051_124_3060_MAR_O_NT_002.SEN3

You will have to repeat this line for all your input data (change the name of files in Ssource), and for the output too, to match the input name_suffix (in my case was “_subset.dim”). You can edit this fast using excel or any other editor that allows copy columns. Once is finished you run it in zhs by writing: sh “myprocess.sh”, being this last file your file containing all the lines that call the gpt for each file to be processed.

Dear marpet,

I download the atmosphericaux data from NASA. also put " “/home/bjiang/snap/anc/”" in the xml, but still does not work. the error in below. Could you give a hand? thanks

Best

Binbin

msi_res20_c2rcc_bin.xml (1.2 KB)

error:
INFO: org.hsqldb.persist.Logger: dataFileCache open start
WARNING: org.esa.s3tbx.c2rcc.ancillary.DataInterpolator: Fallback interpolation start for ozone --> default value = 330.0
WARNING: org.esa.s3tbx.c2rcc.ancillary.DataInterpolator: Fallback interpolation end for ozone --> default value = 330.0
WARNING: org.esa.s3tbx.c2rcc.ancillary.DataInterpolator: Fallback interpolation start for press --> default value = 1000.0
WARNING: org.esa.s3tbx.c2rcc.ancillary.DataInterpolator: Fallback interpolation end for press --> default value = 1000.0

This could have several reasons.
Maybe the auxiliary data has not the correct date or it is not the right format.
Does it work if you specify the products in the GUI, as shown here in the MODIS example?
image
You have downloaded those files? The NCEPR2 and the AUROMI? The NCEPR2 must be unpacked
image