Batch processing Level 2 OLCI scenes

I am trying to batch processing a subset xml graph on L2 S3 images. My subset xml graph and processDataset.bash are attached. I am doing this call:

./processDataset.bash subset_6.xml "/media/silsbelab/S31/S3_L2/S3A_20160216_20190724" "/media/silsbelab/S31/S3_L2/Subset_S3_L2/" subset_S3_L2

I keep getting this error “[NodeId: ReadOp@sourceProduct] Specified ‘file’ [/media/silsbelab/S31/S3_L2/S3A_20160216_20190724/S3A_OL_2_WFR____20160426T144453_20160426T144653_20171030T174540_0119_003_253______MR1_R_NT_002.SEN3:] does not exist.”

There is a semicolon after the file which is strange and I can’t figure out how to delete it. Can someone please let me know if there is something wrong in either my xml graph or bash script?

processDataset.bash (1.5 KB) subset_6.xml (783 Bytes)

I am still having issues with this, if anyone is able to help troubleshoot, I’d greatly appreciate it.

Have you tried first to run from command line one scene? The call should be something like:

gpt_directory xml_directory -t target_scene_directory source_scene_directory

For running a shell you only need to write it and call it wit: sh$

I just tried running one scene from command line using this call:

/home/silsbelab/snap/bin/gpt /media/silsbelab/S31/S3_L2/subset_6.xml -t /media/silsbelab/S31/S3_L2/Subset_S3_L2/ /media/silsbelab/S31/S3_L2/S3A_20160216_20190724/S3A_OL_2_WFR____20160426T144453_20160426T144653_20171030T174540_0119_003_253______MR1_R_NT_002.SEN3

It ran, but made a new folder called which has .hdr and .img files within it. I would like just 1 .nc file that contains everything.

I think you missed to add the name of the output file and the extension in your command after /media/silsbelab/S31/S3_L2/Subset_S3_L2/

great, that worked! Can you help me with the batch processing? I’m confused by how to call it- do I save the call as a .sh file?

How many files you want to process? If not many, just copy the line in a text editor and change the input and output names conveniently. Save as sh (or bat for windows) and run in command line straight forward (
If many, I guess the bash document should work with the for loop over the scenes, but I have never done it. I guess you already know this:

I have 1000+ files I want to subset. I’ve tried using the bash document with the loop, but got the error I described above. This is what I am trying to figure out.

I found an example with some detail here, maybe it can help:

More in:

I added a d to the 1 on line 50 (for F in $(ls -1d "${sourceDirectory}"/*.SEN3); do) and it’s now working. Thanks for your help!

1 Like

I’m now trying to batch process l2gen processed OLCI data using a reproject xml graph using GPT. I have done this successfully with other atmospherically processed data (C2RCC, BAC), however when I run the same scripts on l2gen processed data I get this error:

Executing processing graph
WARNING: org.esa.snap.dataio.netcdf.util.MetadataUtils: Missing configuration property 'snap.dataio.netcdf.metadataElementLimit'. Using default (100).
	at org.esa.snap.core.gpf.common.reproject.ReprojectionOp.getCenterGeoPos(
	at org.esa.snap.core.gpf.common.reproject.ReprojectionOp.initialize(
	at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(
	at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(
	at org.esa.snap.core.gpf.Operator.getTargetProduct(
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(
	at org.esa.snap.core.gpf.graph.GraphContext.initOutput(
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(
	at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(
	at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(
	at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(
	at org.esa.snap.core.gpf.main.GPT.main(
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
	at java.lang.reflect.Method.invoke(
	at org.esa.snap.runtime.Launcher.lambda$run$0(
	at org.esa.snap.runtime.Engine.runClientCode(
	at org.esa.snap.runtime.Launcher.main(
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
	at java.lang.reflect.Method.invoke(
	at com.exe4j.runtime.LauncherEngine.launch(
	at com.install4j.runtime.launcher.UnixLauncher.main(

Error: java.lang.NullPointerException
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters

What does Error: java.lang.NullPointerException mean? I have attached my xml graph, processDataset.bash file. I tried to upload one l2gen processed image but the file size was too large. reproject.xml (1.8 KB) reproject_processDataset.bash (1.5 KB)

It seems that you processed netcdf files are not geocoded. Without this information they cannot be reprojected.
The operator should check the source product and give a more meaningful error message instead of this NullPointerException.

Maybe the data doesn’t have geo-information or it can’t be read by SNAP.
When you try different software, like Panoply or QGIS, do you have proper geo-information?