Error on executing graph

Hi,

I am encountering the following error

Error: [NodeId: ReadOp@sourceProduct] Specified 'file' does not exist

when executing a graph files with gpt. I am quite confused because the when the file name and path are written in .xml file manually it works fine, but when I use ${file} the mentioned error appears.

The input of graph file

<graph id="Graph">
  <version>1.0</version>
  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
        <file>${file1}</file>
    </parameters>
  </node>
  <node id="Read(2)">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
        <file>${file2}</file>
    </parameters>
  </node>
  <node id="TOPSAR-Split">
    <operator>TOPSAR-Split</operator>
    <sources>

and it is executed with the following command

gpt COHERENCE.xml -Pfile1=X:\User\coherence\S1A_IW_SLC__1SDV_20160923T155600_20160923T155627_013182_014F56_24D4.zip -Pfile2=X:\user\coherence\S1A_IW_SLC__1SDV_20161005T155600_20161005T155627_013357_0154DD_F6A4.zip -Ptarget= X:\user\coherence\result\S1A_IW_SLC__1SDV_20160923T155600_20160923T155627_013182_014F56_24D4_coh.dim

What could cause this error?

S.

Could you run your graph with the -e option.
Maybe this will reveal more information about the error.

After running the graph with -e, the result is the following:

Executing processing graph
-- org.jblas INFO Starting temp DLL cleanup task.
-- org.jblas INFO Deleted 8 unused temp DLL libraries from C:\User\AppData\Local\Temp\3
 done.
org.esa.snap.core.gpf.graph.GraphException: [NodeId: ReadOp@sourceProduct] Speci
fied 'file' does not exist
        at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
        at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java
:162)
        at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:91)

        at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:64)

        at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcesso
r.java:130)
        at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(Def
aultCommandLineContext.java:86)
        at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTo
ol.java:534)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.j
ava:388)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(Command
LineTool.java:287)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:1
88)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:1
21)
        at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
        at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
        at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
        at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
        at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:62)
        at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:101)
        at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:26)
Caused by: org.esa.snap.core.gpf.OperatorException: Specified 'file' does not ex
ist
        at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:81)
        at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(Ope
ratorContext.java:486)
        at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(Opera
torContext.java:273)
        at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
        at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext
.java:77)
        ... 28 more

Error: [NodeId: ReadOp@sourceProduct] Specified 'file' does not exist

I have checked the file locations and file path multiple times and everything seems ok. I copy the path and file neme from same source for both cases :

  1. runnig the gpt -Pfile1 and -Pfile2 and ${file1}, ${file2} in .xml
  2. editing the .xml file with the path and file name instead of ${file1}, ${file2}and then run gpt

At the end it works only in case 2.

S.

Mmmh!?
I’m doing it the same way. And it works for me.

I’m using the attached graph file and a S3 SLSTR product . With the following command:

gpt "G:\EOData\Processor Requests\read_SLSTR_reproj_to_UTM.xml" -PslstrFile=G:\EOData\temp\S3A_SL_1_RBT____20160819T200948_20160819T201248_20160819T221500_0179_007_356_2520_MAR_O_NR_001.SEN3\xfdumanifest.xml -t "G:\EOData\temp\test.dim"

From the extended error message I see that the file specified does not exist. But I don’t see which path is checked.
I’ change the error message to include the path of the file. This will be included in an update which will be released in the coming days. Maybe this will bring more insight.

read_SLSTR_reproj_to_UTM.xml (710 Bytes)

Hi Marpet,
I got an error and I feel it is a similar one that skuja got.

 Error: [NodeId: Read] I/O error while reading tie-point grid 'latitude'. 

Could you please have a look? Thank you.
I post also the results after running gpt -e

I see another error message there: “Either operator name or graph XML file must be given”

Dear all,
I got a similar error as Skuja when running S1 input splitting. Was the error ever found / solved?
Regards,
Thijs

Some extra information. I ran the input splitting step in a for loop for all 3 subswaths. Subswath IW1 and subswath IW2 were processed correctly, but when I run the same graph for subswath IW3 I got the same error as Skuja. To be clear: the manifest.safe of the input sentinel image pair is still in the same location as the processing steps for the previous subswaths.

I ran it with gpt -e and got the error message below:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
2020-08-26T13:10:39.977Z SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder ‘.’.
2020-08-26T13:10:41.201Z Executing processing graph
2020-08-26T13:10:42.995Z INFO: org.hsqldb.persist.Logger: dataFileCache open start
2020-08-26T13:10:43.409Z done.
2020-08-26T13:10:43.410Z org.esa.snap.core.gpf.graph.GraphException: [NodeId: Read-Slave] Specified ‘file’ [/home/worker/workDir/inDir/slave/manifest.safe] does not exist.
2020-08-26T13:10:43.412Z at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
2020-08-26T13:10:43.412Z at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
2020-08-26T13:10:43.412Z at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
2020-08-26T13:10:43.413Z at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
2020-08-26T13:10:43.413Z at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
2020-08-26T13:10:43.413Z at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:91)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:64)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
2020-08-26T13:10:43.415Z at java.lang.reflect.Method.invoke(Method.java:498)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
2020-08-26T13:10:43.415Z at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-08-26T13:10:43.415Z at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:62)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-08-26T13:10:43.415Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-08-26T13:10:43.415Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-08-26T13:10:43.415Z at java.lang.reflect.Method.invoke(Method.java:498)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:81)
2020-08-26T13:10:43.416Z … 29 more
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
2020-08-26T13:10:43.416Z Caused by: org.esa.snap.core.gpf.OperatorException: Specified ‘file’ [/home/worker/workDir/inDir/slave/manifest.safe] does not exist.
2020-08-26T13:10:43.417Z Error: [NodeId: Read-Slave] Specified ‘file’ [/home/worker/workDir/inDir/slave/manifest.safe] does not exist.

Any tips?

@ABraun do you have an idea?

very strange… I cannot think of a reason why sub-swath 3 might be treated somehow differently…

@thijs.oosterhuis Does it work when you run the graph from the GUI? Just to go sure that it is not directly related to gpt.

Dear Andreas,
Thanks for the quick reply! Hard to run a for loop from the GUI, but the seperate input split graph does work from the GUI.

Two possible solutions im now trying as well is built in a step to let the VM ‘sleep’ for 60 seconds and copy the sentinel1 image in a seperate input folder for each subswath.

The strangest thing is that it sometimes crashes at IW2, sometimes at IW3, and sometimes not at all.

It is probably not a processing recources issue since it should be similar to each subswath and monitoring it shows still plenty of resources left during processing.

I know these clues probably make it more fuzzy to determine the issue, but I thought it would still might provide extra background info.

Thanks for the help in thinking for a solution anyway!
Thijs

you’re right, sorry - the loop does not work for the GUI…

The fact that it sometimes is IW2 and sometimes IW3 shifts my suspection away from the input data. So it is probably the handling of data streams by the gpt and parameters/input variables are not correctly updated or mixed up.

Did you implement the loop in the command line or with python?

Hey andreas, mo problem!
Im using linux bash on a VM with a ubuntu 16.04 image built as a docker container.
Regards,
Thijs

I’m not very familiar with automating tasks with the Linux bash. Any chance that the iterator (probably sub-swaths 1-3) is falsely inserted into the gpt syntax?
Still would not explain the error message (and why your work around does the job), just thinking of possible error sources.

Dear Andreas,
I also dont expect this to be the issue, since my snap xml is created with these parameters:

SNAP_REQUEST= /home/worker/processor/create_snap_request_split.sh master=/home/worker/workDir/inDir/master/manifest.safe slave=/home/worker/workDir/inDir/slave/manifest.safe cursubswath=IW3 pol=VV outmastersplit=/home/worker/procDir/split_product/target_IW3_VV_Split_Master outslavesplit=/home/worker/procDir/split_product/target_IW3_VV_Split_Slave

All these parameters are what i expect them to be and the location of the manifest.safe is the same as it was for IW2 and IW1 where it worked correcly.

Hi!

Its been a while since the error that I encountered but as far as I remember I reinstalled the SNAP and checked permission settings for the folders where the Sentinel files were stored. Eventually, I was able to run the process.

Best,
S

1 Like

Thanks for the update Skuja! This would not help me since im working in a docker container.
Its probably a different issue im facing then! Might even be unrelated to snap…
Regards,
Thijs

Hello guys.

I’m facing a similar issue when running my graph:

(base) [ab302@rcglogin02 download_S2]$ gpt graph1.xml -e -t here.dim S2_2017/S2A_MSIL1C_20170101T235242_N0204_R130_T56HLJ_20170101T235308.SAFE                                       INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32756
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
 done.
org.esa.snap.core.gpf.graph.GraphException: [NodeId: Read] Specified 'file' [ReadOp@sourceProduct] does not exist.
        at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
        at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
        at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
        at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:85)
        at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:58)
        at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
        at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
        at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
        at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
        at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
        at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
        at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
        at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
        at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
        at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
        at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
        at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: Specified 'file' [ReadOp@sourceProduct] does not exist.
        at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:113)
        at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
        at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
        at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
        at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
        ... 32 more

Error: [NodeId: Read] Specified 'file' [ReadOp@sourceProduct] does not exist.
(base) [ab302@rcglogin02 download_S2]$

The folder S2_2017 has the following rights: drwxr-xr-x, and so all S2*.SAFE directories inside it.

I am running in a computer cluster from the University here, thus, I am wondering if it might have to do with admin rights.

The content of graph1.xml is as follows:

<graph id="graph">
    <version>1.0</version>
    
    <node id="Read">
        <operator>Read</operator>
        <sources/>
        <parameters>
            <file>${sourceProduct}</file>
        </parameters>
    </node>
    
    <node id="resample-s2">
        <operator>Resample</operator>
        <sources>
            <sourceProduct refid="Read"/>
        </sources>
        <parameters>
            <targetResolution>20</targetResolution>
            <upsampling>Bilinear</upsampling>
            <downsampling>Mean</downsampling>
            <flagDownsampling>First</flagDownsampling>
            <resampleOnPyramidLevels>false</resampleOnPyramidLevels>
        </parameters>
    </node>
    
    <node id="c2rcc">
        <operator>c2rcc.msi</operator>
        <sources>
            <sourceProduct refid="resample-s2"/>
        </sources>
        <parameters>
            <salinity>35</salinity>
            <temperature>15</temperature>
        </parameters>
    </node>
    
    <!-- <node id="TSMonly">
        <operator>Subset</operator>
        <sources>
            <sourceProduct refid="c2rcc"/>
        </sources>
        <parameters>
            <sourceBands>conc_tsm</sourceBands>
            <copyMetadata>false</copyMetadata>
        </parameters>
    </node> -->
	
	<node id="TSMonly">
        <operator>BandMaths</operator>
		<sources>
		    <sourceProducts>c2rcc</sourceProducts>
		</sources>
		<parameters>
		    <targetBands>
			    <targetBand>
				    <name>tsm_conc</name>
                    <type>float32</type>
                    <expression>conc_tsm</expression>
                    <description>Total suspended matter dry weight concentration</description>
                    <unit>g m^-3</unit>
                </targetBand>
            </targetBands>
        </parameters>
    </node>
    
</graph>

Any light on that?

That’s a mistake in the configuration.

If you invoke gpt like,
gpt ... S2_2017/S2A_MSIL1C_20170101T235242_N0204_R130_T56HLJ_20170101T235308.SAFE
Then the source product is automatically loaded, and the read operator is not necessary.

If you want to keep the ReadOp then you can turn the source product into a parameter.
gpt ... -PinProduct=S2_2017/S2A_MSIL1C_20170101T235242_N0204_R130_T56HLJ_20170101T235308.SAFE

In the xml you can use the parameter like this:

    <node id="Read">
        <operator>Read</operator>
        <sources/>
        <parameters>
            <file>${inProduct}</file>
        </parameters>
    </node>

If you keep the gpt call, then remove the read node and change the Resample node to

<node id="resample-s2">
        <operator>Resample</operator>
        <sources>
            <sourceProduct>${sourceProduct}</sourceProduct>
        </sources>
        <parameters>
...

Yeah, that’s it.

Working fine now.

Thank you, Marco!

1 Like