Batch processing Level 2 OLCI scenes

Optical Toolbox
#1

I am trying to batch processing a subset xml graph on L2 S3 images. My subset xml graph and processDataset.bash are attached. I am doing this call:

./processDataset.bash subset_6.xml parameters.properties "/media/silsbelab/S31/S3_L2/S3A_20160216_20190724" "/media/silsbelab/S31/S3_L2/Subset_S3_L2/" subset_S3_L2

I keep getting this error “[NodeId: ReadOp@sourceProduct] Specified ‘file’ [/media/silsbelab/S31/S3_L2/S3A_20160216_20190724/S3A_OL_2_WFR____20160426T144453_20160426T144653_20171030T174540_0119_003_253______MR1_R_NT_002.SEN3:] does not exist.”

There is a semicolon after the file which is strange and I can’t figure out how to delete it. Can someone please let me know if there is something wrong in either my xml graph or bash script?

processDataset.bash (1.5 KB) subset_6.xml (783 Bytes)

#2

I am still having issues with this, if anyone is able to help troubleshoot, I’d greatly appreciate it.

#3

Have you tried first to run from command line one scene? The call should be something like:

gpt_directory xml_directory -t target_scene_directory source_scene_directory

For running a shell you only need to write it and call it wit: sh$ name.sh

#4

I just tried running one scene from command line using this call:

/home/silsbelab/snap/bin/gpt /media/silsbelab/S31/S3_L2/subset_6.xml -t /media/silsbelab/S31/S3_L2/Subset_S3_L2/ /media/silsbelab/S31/S3_L2/S3A_20160216_20190724/S3A_OL_2_WFR____20160426T144453_20160426T144653_20171030T174540_0119_003_253______MR1_R_NT_002.SEN3

It ran, but made a new folder called Subset_S3_L2.data which has .hdr and .img files within it. I would like just 1 .nc file that contains everything.

#5

I think you missed to add the name of the output file and the extension in your command after /media/silsbelab/S31/S3_L2/Subset_S3_L2/name_file.nc

#6

great, that worked! Can you help me with the batch processing? I’m confused by how to call it- do I save the call as a .sh file?

#7

How many files you want to process? If not many, just copy the line in a text editor and change the input and output names conveniently. Save as sh (or bat for windows) and run in command line straight forward (my_file.sh)
If many, I guess the bash document should work with the for loop over the scenes, but I have never done it. I guess you already know this:
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

#8

I have 1000+ files I want to subset. I’ve tried using the bash document with the loop, but got the error I described above. This is what I am trying to figure out.

#9

I found an example with some detail here, maybe it can help:

More in:

#10

I added a d to the 1 on line 50 (for F in $(ls -1d "${sourceDirectory}"/*.SEN3); do) and it’s now working. Thanks for your help!

1 Like
#13

I’m now trying to batch process l2gen processed OLCI data using a reproject xml graph using GPT. I have done this successfully with other atmospherically processed data (C2RCC, BAC), however when I run the same scripts on l2gen processed data I get this error:

Executing processing graph
WARNING: org.esa.snap.dataio.netcdf.util.MetadataUtils: Missing configuration property 'snap.dataio.netcdf.metadataElementLimit'. Using default (100).
 done.
java.lang.NullPointerException
	at org.esa.snap.core.gpf.common.reproject.ReprojectionOp.getCenterGeoPos(ReprojectionOp.java:584)
	at org.esa.snap.core.gpf.common.reproject.ReprojectionOp.initialize(ReprojectionOp.java:239)
	at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
	at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
	at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
	at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:91)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:64)
	at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)
	at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
	at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
	at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
	at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
	at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
	at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
	at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
	at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

Error: java.lang.NullPointerException
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters

What does Error: java.lang.NullPointerException mean? I have attached my xml graph, processDataset.bash file. I tried to upload one l2gen processed image but the file size was too large. reproject.xml (1.8 KB) reproject_processDataset.bash (1.5 KB)

#14

It seems that you processed netcdf files are not geocoded. Without this information they cannot be reprojected.
The operator should check the source product and give a more meaningful error message instead of this NullPointerException.

Maybe the data doesn’t have geo-information or it can’t be read by SNAP.
When you try different software, like Panoply or QGIS, do you have proper geo-information?