Error: [NodeId: ReadOp@sourceProduct] Specified 'file' does not exist
when executing a graph files with gpt. I am quite confused because the when the file name and path are written in .xml file manually it works fine, but when I use ${file} the mentioned error appears.
After running the graph with -e, the result is the following:
Executing processing graph
-- org.jblas INFO Starting temp DLL cleanup task.
-- org.jblas INFO Deleted 8 unused temp DLL libraries from C:\User\AppData\Local\Temp\3
done.
org.esa.snap.core.gpf.graph.GraphException: [NodeId: ReadOp@sourceProduct] Speci
fied 'file' does not exist
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java
:162)
at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:91)
at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:64)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcesso
r.java:130)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(Def
aultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTo
ol.java:534)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.j
ava:388)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(Command
LineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:1
88)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:1
21)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:62)
at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:101)
at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:26)
Caused by: org.esa.snap.core.gpf.OperatorException: Specified 'file' does not ex
ist
at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:81)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(Ope
ratorContext.java:486)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(Opera
torContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext
.java:77)
... 28 more
Error: [NodeId: ReadOp@sourceProduct] Specified 'file' does not exist
I have checked the file locations and file path multiple times and everything seems ok. I copy the path and file neme from same source for both cases :
runnig the gpt -Pfile1 and -Pfile2 and ${file1}, ${file2} in .xml
editing the .xml file with the path and file name instead of ${file1}, ${file2}and then run gpt
From the extended error message I see that the file specified does not exist. But I don’t see which path is checked.
I’ change the error message to include the path of the file. This will be included in an update which will be released in the coming days. Maybe this will bring more insight.
Some extra information. I ran the input splitting step in a for loop for all 3 subswaths. Subswath IW1 and subswath IW2 were processed correctly, but when I run the same graph for subswath IW3 I got the same error as Skuja. To be clear: the manifest.safe of the input sentinel image pair is still in the same location as the processing steps for the previous subswaths.
I ran it with gpt -e and got the error message below:
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
2020-08-26T13:10:39.977Z SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder ‘.’.
2020-08-26T13:10:41.201Z Executing processing graph
2020-08-26T13:10:42.995Z INFO: org.hsqldb.persist.Logger: dataFileCache open start
2020-08-26T13:10:43.409Z done.
2020-08-26T13:10:43.410Z org.esa.snap.core.gpf.graph.GraphException: [NodeId: Read-Slave] Specified ‘file’ [/home/worker/workDir/inDir/slave/manifest.safe] does not exist.
2020-08-26T13:10:43.412Z at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
2020-08-26T13:10:43.412Z at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
2020-08-26T13:10:43.412Z at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
2020-08-26T13:10:43.413Z at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
2020-08-26T13:10:43.413Z at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
2020-08-26T13:10:43.413Z at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:91)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:64)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
2020-08-26T13:10:43.414Z at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
2020-08-26T13:10:43.415Z at java.lang.reflect.Method.invoke(Method.java:498)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
2020-08-26T13:10:43.415Z at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-08-26T13:10:43.415Z at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:62)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-08-26T13:10:43.415Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-08-26T13:10:43.415Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
2020-08-26T13:10:43.415Z at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
2020-08-26T13:10:43.415Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-08-26T13:10:43.415Z at java.lang.reflect.Method.invoke(Method.java:498)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:81)
2020-08-26T13:10:43.416Z … 29 more
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
2020-08-26T13:10:43.416Z at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
2020-08-26T13:10:43.416Z Caused by: org.esa.snap.core.gpf.OperatorException: Specified ‘file’ [/home/worker/workDir/inDir/slave/manifest.safe] does not exist.
2020-08-26T13:10:43.417Z Error: [NodeId: Read-Slave] Specified ‘file’ [/home/worker/workDir/inDir/slave/manifest.safe] does not exist.
Dear Andreas,
Thanks for the quick reply! Hard to run a for loop from the GUI, but the seperate input split graph does work from the GUI.
Two possible solutions im now trying as well is built in a step to let the VM ‘sleep’ for 60 seconds and copy the sentinel1 image in a seperate input folder for each subswath.
The strangest thing is that it sometimes crashes at IW2, sometimes at IW3, and sometimes not at all.
It is probably not a processing recources issue since it should be similar to each subswath and monitoring it shows still plenty of resources left during processing.
I know these clues probably make it more fuzzy to determine the issue, but I thought it would still might provide extra background info.
Thanks for the help in thinking for a solution anyway!
Thijs
you’re right, sorry - the loop does not work for the GUI…
The fact that it sometimes is IW2 and sometimes IW3 shifts my suspection away from the input data. So it is probably the handling of data streams by the gpt and parameters/input variables are not correctly updated or mixed up.
Did you implement the loop in the command line or with python?
I’m not very familiar with automating tasks with the Linux bash. Any chance that the iterator (probably sub-swaths 1-3) is falsely inserted into the gpt syntax?
Still would not explain the error message (and why your work around does the job), just thinking of possible error sources.
All these parameters are what i expect them to be and the location of the manifest.safe is the same as it was for IW2 and IW1 where it worked correcly.
Its been a while since the error that I encountered but as far as I remember I reinstalled the SNAP and checked permission settings for the folders where the Sentinel files were stored. Eventually, I was able to run the process.
Thanks for the update Skuja! This would not help me since im working in a docker container.
Its probably a different issue im facing then! Might even be unrelated to snap…
Regards,
Thijs
(base) [ab302@rcglogin02 download_S2]$ gpt graph1.xml -e -t here.dim S2_2017/S2A_MSIL1C_20170101T235242_N0204_R130_T56HLJ_20170101T235308.SAFE INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32756
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
done.
org.esa.snap.core.gpf.graph.GraphException: [NodeId: Read] Specified 'file' [ReadOp@sourceProduct] does not exist.
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:85)
at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:58)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: Specified 'file' [ReadOp@sourceProduct] does not exist.
at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:113)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
... 32 more
Error: [NodeId: Read] Specified 'file' [ReadOp@sourceProduct] does not exist.
(base) [ab302@rcglogin02 download_S2]$
The folder S2_2017 has the following rights: drwxr-xr-x, and so all S2*.SAFE directories inside it.
I am running in a computer cluster from the University here, thus, I am wondering if it might have to do with admin rights.
If you invoke gpt like, gpt ... S2_2017/S2A_MSIL1C_20170101T235242_N0204_R130_T56HLJ_20170101T235308.SAFE
Then the source product is automatically loaded, and the read operator is not necessary.
If you want to keep the ReadOp then you can turn the source product into a parameter. gpt ... -PinProduct=S2_2017/S2A_MSIL1C_20170101T235242_N0204_R130_T56HLJ_20170101T235308.SAFE