Can't apply C2RCC in S2 images due to 'iop_bpart' undefined symbol

Hello! About two years ago I successfully used a set of batch files to extract TSM data from raw S2 images.

Over the last month I downloaded a bunch of S2 images from 2017 and today I tried to run the same script but I got the following error for all of them:

D:\TSM>(
echo.
set sourceFile=D:\TSM\2017-S2MSI\S2A_MSIL1C_20170111T235231_N0204_R130_T56HLJ_20170111T235335.SAFE
echo Processing !sourceFile!
set targetFile=.\2017-S2MSI\tsm\c2rcc_tsm_S2A_MSIL1C_20170111T235231_N0204_R130_T56HLJ_20170111T235335.dim
set procCmd=“C:\Program Files\snap\bin\gpt.exe” .\graph1.xml -e -t “!targetFile!” -PsourceProduct="!sourceFile!"
call !procCmd!
)

Processing D:\TSM\2017-S2MSI\S2A_MSIL1C_20170111T235231_N0204_R130_T56HLJ_20170111T235335.SAFE
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32756
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
.INFO: org.esa.s3tbx.c2rcc.ancillary.AtmosphericAuxdataBuilder: Atmospheric auxdata product can’t be used. At least one is not specified. Using constant values for ozone (330.0) and surface pressure (1000.0).
.90% done.
java.lang.IllegalArgumentException: expression: Undefined symbol ‘iop_bpart’.
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:108)
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:94)
at org.esa.snap.core.datamodel.VirtualBand.createSourceImage(VirtualBand.java:264)
at org.esa.snap.core.datamodel.VirtualBand.createSourceImage(VirtualBand.java:247)
at org.esa.snap.core.datamodel.RasterDataNode.getSourceImage(RasterDataNode.java:2174)
at org.esa.snap.core.gpf.common.WriteOp.doExecute(WriteOp.java:282)
at org.esa.snap.core.gpf.internal.OperatorContext.executeOperator(OperatorContext.java:1300)
at org.esa.snap.core.gpf.Operator.execute(Operator.java:153)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:189)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:119)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:60)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:94)
at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:25)
Caused by: org.esa.snap.core.jexp.ParseException: Undefined symbol ‘iop_bpart’.
at org.esa.snap.core.jexp.impl.ParserImpl.reportError(ParserImpl.java:1036)
at org.esa.snap.core.jexp.impl.ParserImpl.parseCallOrRef(ParserImpl.java:722)
at org.esa.snap.core.jexp.impl.ParserImpl.parsePrimary(ParserImpl.java:678)
at org.esa.snap.core.jexp.impl.ParserImpl.parsePostfix(ParserImpl.java:637)
at org.esa.snap.core.jexp.impl.ParserImpl.parseUnary(ParserImpl.java:624)
at org.esa.snap.core.jexp.impl.ParserImpl.parseMul(ParserImpl.java:530)
at org.esa.snap.core.jexp.impl.ParserImpl.parseAdd(ParserImpl.java:487)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseAnd(ParserImpl.java:461)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseXOr(ParserImpl.java:434)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseOr(ParserImpl.java:408)
at org.esa.snap.core.jexp.impl.ParserImpl.parseComparison(ParserImpl.java:319)
at org.esa.snap.core.jexp.impl.ParserImpl.parseLogicalAnd(ParserImpl.java:286)
at org.esa.snap.core.jexp.impl.ParserImpl.parseLogicalOr(ParserImpl.java:259)
at org.esa.snap.core.jexp.impl.ParserImpl.parseConditional(ParserImpl.java:230)
at org.esa.snap.core.jexp.impl.ParserImpl.parseAssign(ParserImpl.java:179)
at org.esa.snap.core.jexp.impl.ParserImpl.parseTerm(ParserImpl.java:168)
at org.esa.snap.core.jexp.impl.ParserImpl.parseImpl(ParserImpl.java:150)
at org.esa.snap.core.jexp.impl.ParserImpl.parse(ParserImpl.java:136)
at org.esa.snap.core.jexp.impl.ParserImpl.parse(ParserImpl.java:115)
at org.esa.snap.core.dataop.barithm.BandArithmetic.parseExpression(BandArithmetic.java:116)
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:106)
… 32 more

Error: expression: Undefined symbol ‘iop_bpart’.

To execute the extraction I have 3 files which I call using the following syntax:
.\full_process.bat .\graph1.xml .\input_folder\ .\output_folder c2rcc_tsm .\graph2.xml

Please find below the content of each one of these three files:

<full_process.bat>

::@echo off
:: enable delayed expansion - used to resolve variable in loop
:: variable has to be used with '!' instead of '%'
setlocal ENABLEDELAYEDEXPANSION

:: USING EXAMPLE
::> batchfile C2RCC_graph.xml folder_with_input_files folder_to_save_outputs c2rcc_prefix PixelExtrac.xml
::> .\full_process.bat .\graph1.xml .\sentinel2\ .\sentinel2\tsm c2rcc_tsm .\graph2.xml

::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::

:: adapt this path to your needs
set gptPath="C:\Program Files\snap\bin\gpt.exe"

::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling for first graph
::::::::::::::::::::::::::::::::::::::::::::

:: first parameter is a path to the graph xml
set graphXmlPath=%1

:: use second parameter for path to source products
set sourceDirectory=%2
:: if sourceDirectory ends with '\' remove it
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%

:: use third parameter for path to target products
set targetDirectory=%3
:: if targetDirectory ends with '\' remove it
if %targetDirectory:~-1%==\ set targetDirectory=%targetDirectory:~0,-1%

:: the fourth parameter is a file prefix for the target product name, 
:: typically indicating the type of processing
set targetFilePrefix=%4

:: Create the target directory
md %targetDirectory%

::::::::::::::::::::::::::::::::::::::::::::
:: Main processing of first graph
::::::::::::::::::::::::::::::::::::::::::::

:: double '%' in batch file and only a single '%' on command line
:: '/D' is for directories like Sentinel data. Remove '/D' when you open files.
:: Sentinel 2 MSI = (S2*.SAFE)
:: Sentinel 3 OLCI = (S3*.SEN3)
for /D /R %sourceDirectory% %%F in (S2*.SAFE) do (
  echo.
  :: '~fF' means abolute path of 'F'
  set sourceFile=%%~fF
  echo Processing !sourceFile!
  :: '~nF' means filename without extension of 'F'
  set targetFile=%targetDirectory%\%targetFilePrefix%_%%~nF.dim
  set procCmd=%gptPath% %graphXmlPath% -e -t "!targetFile!" -PsourceProduct="!sourceFile!"
  call !procCmd!
)


::::::::::::::::::::::::::::::::::::::::::::
:: Processing second graph
::::::::::::::::::::::::::::::::::::::::::::
:: fifth parameter is a path to the graph xml
set graphXmlPath=%5

:: selection of BEAM-DIM files with TSM data
set inputFiles=%targetDirectory%\**.dim
echo TSM input files: !inputFiles!

set procCmd=%gptPath% %graphXmlPath% -e -PsourceProductPaths="!inputFiles!" -PoutputDir="!targetDirectory!"
call !procCmd!

<graph1.xml>

<graph id="graph">
    <version>1.0</version>
    
    <node id="Read">
        <operator>Read</operator>
        <sources/>
        <parameters>
            <file>${sourceProduct}</file>
        </parameters>
    </node>
    
    <node id="resample-s2">
        <operator>Resample</operator>
        <sources>
            <sourceProduct refid="Read"/>
        </sources>
        <parameters>
            <targetResolution>20</targetResolution>
            <upsampling>Bilinear</upsampling>
            <downsampling>Mean</downsampling>
            <flagDownsampling>First</flagDownsampling>
            <resampleOnPyramidLevels>false</resampleOnPyramidLevels>
        </parameters>
    </node>
    
    <node id="c2rcc">
        <operator>c2rcc.msi</operator>
        <sources>
            <sourceProduct refid="resample-s2"/>
        </sources>
        <parameters>
            <salinity>35</salinity>
            <temperature>15</temperature>
        </parameters>
    </node>
    
    <node id="TSMonly">
        <operator>Subset</operator>
        <sources>
            <sourceProduct refid="c2rcc"/>
        </sources>
        <parameters>
            <sourceBands>conc_tsm</sourceBands>
            <copyMetadata>false</copyMetadata>
        </parameters>
    </node>
        
</graph>

<graph2.xml>

<graph id="point_extraction">
    <version>1.0</version>
    
    <node id="tsm_values">
        <operator>PixEx</operator>
        <parameters>
            <sourceProductPaths>${sourceProductPaths}</sourceProductPaths>
            <exportBands>true</exportBands>
            <exportTiePoints>false</exportTiePoints>
            <exportMasks>false</exportMasks>
            <coordinatesFile>C:\Wetlands\TSM\pins_hunter_estuary.placemark</coordinatesFile>
            <windowSize>1</windowSize>
            <outputDir>${outputDir}</outputDir>
            <outputFilePrefix>2017</outputFilePrefix>
            <extractTimeFromFilename>false</extractTimeFromFilename>
        </parameters>
    </node>
    
</graph>

Interestingly, I can execute the resampling and C2RCC procedures using the desktop version for SNAP 8. However, I can do only one image at once.

So, basically, did something change from SNAP 7 to SNAP 8 that I should review in any of the XML or BAT files?

Secondly. Is there a way to resample just a piece of the image, like between the coordinates of the bottom-left corner and the top-right one? This would save a lot of processing time for me.

Kind regards,
Angelo.

1 Like

Hello Angelo,

limiting the data to the conc_tsm band causes the problem, I guess.
It is a virtual band and needs as input the band iop_btot, and this band needs iop_bpart and iop_bwit.

If you include those three additional bands, it will probably work. Maybe you must include also the c2rcc_flags band.

To have only the con_tsm in the output you can replace the subset step by a band math step.
This could look like:

 <node id="TSMonly">
      <operator>BandMaths</operator>
      <sources>
        <sourceProducts>c2rcc</sourceProducts>
      </sources>
      <parameters>
        <targetBands>
          <targetBand>
            <name>tsm_conc</name>
            <type>float32</type>
            <expression>conc_tsm</expression>
            <description>Total suspended matter dry weight concentration</description>
            <unit>g m^-3</unit>
          </targetBand>
          <.../>
        </targetBands>
      </parameters>
    </node>

This creates a new band ‘tsm_conc’ and the references to the other bands are resolved.

The definition of the conc_tsm has be changed in SNAP 8:
[SIIITBX-379] Expression and factors for TSM calculation need to be updated for S2-MSI C2RCC
What still puzzles me is that it worked for you in SNAP 7. Because in this version it contained references to iop_bpart and iop_bwit. So you should have seen similar problem.

Hi Marco,

The batch process is now working fine! I just replaced this code snippet over my TSMonly node in graph1.xml (and removed the line “<…/>”), anything else.

I am actually not sure if it was SNAP 7 that I was using back in 2019, maybe it was a previous version.

Thank you very much!

1 Like