Hello! About two years ago I successfully used a set of batch files to extract TSM data from raw S2 images.
Over the last month I downloaded a bunch of S2 images from 2017 and today I tried to run the same script but I got the following error for all of them:
D:\TSM>(
echo.
set sourceFile=D:\TSM\2017-S2MSI\S2A_MSIL1C_20170111T235231_N0204_R130_T56HLJ_20170111T235335.SAFE
echo Processing !sourceFile!
set targetFile=.\2017-S2MSI\tsm\c2rcc_tsm_S2A_MSIL1C_20170111T235231_N0204_R130_T56HLJ_20170111T235335.dim
set procCmd=“C:\Program Files\snap\bin\gpt.exe” .\graph1.xml -e -t “!targetFile!” -PsourceProduct=“!sourceFile!”
call !procCmd!
)Processing D:\TSM\2017-S2MSI\S2A_MSIL1C_20170111T235231_N0204_R130_T56HLJ_20170111T235335.SAFE
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32756
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
.INFO: org.esa.s3tbx.c2rcc.ancillary.AtmosphericAuxdataBuilder: Atmospheric auxdata product can’t be used. At least one is not specified. Using constant values for ozone (330.0) and surface pressure (1000.0).
.90% done.
java.lang.IllegalArgumentException: expression: Undefined symbol ‘iop_bpart’.
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:108)
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:94)
at org.esa.snap.core.datamodel.VirtualBand.createSourceImage(VirtualBand.java:264)
at org.esa.snap.core.datamodel.VirtualBand.createSourceImage(VirtualBand.java:247)
at org.esa.snap.core.datamodel.RasterDataNode.getSourceImage(RasterDataNode.java:2174)
at org.esa.snap.core.gpf.common.WriteOp.doExecute(WriteOp.java:282)
at org.esa.snap.core.gpf.internal.OperatorContext.executeOperator(OperatorContext.java:1300)
at org.esa.snap.core.gpf.Operator.execute(Operator.java:153)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:189)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:119)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:60)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:94)
at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:25)
Caused by: org.esa.snap.core.jexp.ParseException: Undefined symbol ‘iop_bpart’.
at org.esa.snap.core.jexp.impl.ParserImpl.reportError(ParserImpl.java:1036)
at org.esa.snap.core.jexp.impl.ParserImpl.parseCallOrRef(ParserImpl.java:722)
at org.esa.snap.core.jexp.impl.ParserImpl.parsePrimary(ParserImpl.java:678)
at org.esa.snap.core.jexp.impl.ParserImpl.parsePostfix(ParserImpl.java:637)
at org.esa.snap.core.jexp.impl.ParserImpl.parseUnary(ParserImpl.java:624)
at org.esa.snap.core.jexp.impl.ParserImpl.parseMul(ParserImpl.java:530)
at org.esa.snap.core.jexp.impl.ParserImpl.parseAdd(ParserImpl.java:487)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseAnd(ParserImpl.java:461)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseXOr(ParserImpl.java:434)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseOr(ParserImpl.java:408)
at org.esa.snap.core.jexp.impl.ParserImpl.parseComparison(ParserImpl.java:319)
at org.esa.snap.core.jexp.impl.ParserImpl.parseLogicalAnd(ParserImpl.java:286)
at org.esa.snap.core.jexp.impl.ParserImpl.parseLogicalOr(ParserImpl.java:259)
at org.esa.snap.core.jexp.impl.ParserImpl.parseConditional(ParserImpl.java:230)
at org.esa.snap.core.jexp.impl.ParserImpl.parseAssign(ParserImpl.java:179)
at org.esa.snap.core.jexp.impl.ParserImpl.parseTerm(ParserImpl.java:168)
at org.esa.snap.core.jexp.impl.ParserImpl.parseImpl(ParserImpl.java:150)
at org.esa.snap.core.jexp.impl.ParserImpl.parse(ParserImpl.java:136)
at org.esa.snap.core.jexp.impl.ParserImpl.parse(ParserImpl.java:115)
at org.esa.snap.core.dataop.barithm.BandArithmetic.parseExpression(BandArithmetic.java:116)
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:106)
… 32 moreError: expression: Undefined symbol ‘iop_bpart’.
To execute the extraction I have 3 files which I call using the following syntax:
.\full_process.bat .\graph1.xml .\input_folder\ .\output_folder c2rcc_tsm .\graph2.xml
Please find below the content of each one of these three files:
<full_process.bat>
::@echo off
:: enable delayed expansion - used to resolve variable in loop
:: variable has to be used with '!' instead of '%'
setlocal ENABLEDELAYEDEXPANSION
:: USING EXAMPLE
::> batchfile C2RCC_graph.xml folder_with_input_files folder_to_save_outputs c2rcc_prefix PixelExtrac.xml
::> .\full_process.bat .\graph1.xml .\sentinel2\ .\sentinel2\tsm c2rcc_tsm .\graph2.xml
::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::
:: adapt this path to your needs
set gptPath="C:\Program Files\snap\bin\gpt.exe"
::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling for first graph
::::::::::::::::::::::::::::::::::::::::::::
:: first parameter is a path to the graph xml
set graphXmlPath=%1
:: use second parameter for path to source products
set sourceDirectory=%2
:: if sourceDirectory ends with '\' remove it
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%
:: use third parameter for path to target products
set targetDirectory=%3
:: if targetDirectory ends with '\' remove it
if %targetDirectory:~-1%==\ set targetDirectory=%targetDirectory:~0,-1%
:: the fourth parameter is a file prefix for the target product name,
:: typically indicating the type of processing
set targetFilePrefix=%4
:: Create the target directory
md %targetDirectory%
::::::::::::::::::::::::::::::::::::::::::::
:: Main processing of first graph
::::::::::::::::::::::::::::::::::::::::::::
:: double '%' in batch file and only a single '%' on command line
:: '/D' is for directories like Sentinel data. Remove '/D' when you open files.
:: Sentinel 2 MSI = (S2*.SAFE)
:: Sentinel 3 OLCI = (S3*.SEN3)
for /D /R %sourceDirectory% %%F in (S2*.SAFE) do (
echo.
:: '~fF' means abolute path of 'F'
set sourceFile=%%~fF
echo Processing !sourceFile!
:: '~nF' means filename without extension of 'F'
set targetFile=%targetDirectory%\%targetFilePrefix%_%%~nF.dim
set procCmd=%gptPath% %graphXmlPath% -e -t "!targetFile!" -PsourceProduct="!sourceFile!"
call !procCmd!
)
::::::::::::::::::::::::::::::::::::::::::::
:: Processing second graph
::::::::::::::::::::::::::::::::::::::::::::
:: fifth parameter is a path to the graph xml
set graphXmlPath=%5
:: selection of BEAM-DIM files with TSM data
set inputFiles=%targetDirectory%\**.dim
echo TSM input files: !inputFiles!
set procCmd=%gptPath% %graphXmlPath% -e -PsourceProductPaths="!inputFiles!" -PoutputDir="!targetDirectory!"
call !procCmd!
<graph1.xml>
<graph id="graph">
<version>1.0</version>
<node id="Read">
<operator>Read</operator>
<sources/>
<parameters>
<file>${sourceProduct}</file>
</parameters>
</node>
<node id="resample-s2">
<operator>Resample</operator>
<sources>
<sourceProduct refid="Read"/>
</sources>
<parameters>
<targetResolution>20</targetResolution>
<upsampling>Bilinear</upsampling>
<downsampling>Mean</downsampling>
<flagDownsampling>First</flagDownsampling>
<resampleOnPyramidLevels>false</resampleOnPyramidLevels>
</parameters>
</node>
<node id="c2rcc">
<operator>c2rcc.msi</operator>
<sources>
<sourceProduct refid="resample-s2"/>
</sources>
<parameters>
<salinity>35</salinity>
<temperature>15</temperature>
</parameters>
</node>
<node id="TSMonly">
<operator>Subset</operator>
<sources>
<sourceProduct refid="c2rcc"/>
</sources>
<parameters>
<sourceBands>conc_tsm</sourceBands>
<copyMetadata>false</copyMetadata>
</parameters>
</node>
</graph>
<graph2.xml>
<graph id="point_extraction">
<version>1.0</version>
<node id="tsm_values">
<operator>PixEx</operator>
<parameters>
<sourceProductPaths>${sourceProductPaths}</sourceProductPaths>
<exportBands>true</exportBands>
<exportTiePoints>false</exportTiePoints>
<exportMasks>false</exportMasks>
<coordinatesFile>C:\Wetlands\TSM\pins_hunter_estuary.placemark</coordinatesFile>
<windowSize>1</windowSize>
<outputDir>${outputDir}</outputDir>
<outputFilePrefix>2017</outputFilePrefix>
<extractTimeFromFilename>false</extractTimeFromFilename>
</parameters>
</node>
</graph>
Interestingly, I can execute the resampling and C2RCC procedures using the desktop version for SNAP 8. However, I can do only one image at once.
So, basically, did something change from SNAP 7 to SNAP 8 that I should review in any of the XML or BAT files?
Secondly. Is there a way to resample just a piece of the image, like between the coordinates of the bottom-left corner and the top-right one? This would save a lot of processing time for me.
Kind regards,
Angelo.