Help applying mask/filter before algorithm with GPT

OliGraffeuille · October 20, 2020, 7:50am

Hi there,

Apologies if this has already been discussed, I couldn’t find any similar discussion.

I am wanting to apply the RayleighCorrection algorithm to S3 OLCI L1b products. However, I am only interested in fresh water pixels, that is, pixels where the L1b quality flag “FRESH_INLAND_WATER” is true. Since the RayleighCorrection algorithm is somewhat computationally expensive, I am trying to find a way to apply the RayleighCorrection algorithm to FRESH_INLAND_WATER pixels.

To clarify, I’d ideally want something like the validPixelExpression parameter that exists the C2RCC algorithm.

I’ve tried creating a gpt .xml file with three nodes: Read -> Land-Sea-Mask -> RayleighCorrection. The file is attached here RayleighCorrectionGraph.xml (1.2 KB) .

This gpt returns the following error when running, however:

C:\Users\***>gpt D:\TAIAO\temp\BRR2.xml -t D:\TAIAO\temp\BRR2_output -f 'Net
CDF4-CF' -PsourceProduct=E:\sentinel-3_data\gpt_test\S3A_OL_1_EFR____20200130T21
2751_20200130T213051_20200201T005857_0179_054_214_3600_LN1_O_NT_002.SEN3
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing ex
ternal tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly
 for new updates for the best SNAP experience.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-c
si-srtm.openterrain.org.s3.amazonaws.com/source/srtm_72_19.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_72_19.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_72_19.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.cs
i.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_72_19.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://srtm.csi.
cgiar.org/wp-content/uploads/files/srtm_5x5/tiff/srtm_72_19.zip on http://srtm.c
si.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_72_19.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-c
si-srtm.openterrain.org.s3.amazonaws.com/source/srtm_72_19.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_72_19.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_72_19.zip
...INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgia
r-csi-srtm.openterrain.org.s3.amazonaws.com/source/srtm_01_20.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_01_20.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_01_20.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.cs
i.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_01_20.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://srtm.csi.
cgiar.org/wp-content/uploads/files/srtm_5x5/tiff/srtm_01_20.zip on http://srtm.c
si.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_01_20.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-c
si-srtm.openterrain.org.s3.amazonaws.com/source/srtm_01_20.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_01_20.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_01_20.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-c
si-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_19.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_19.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_19.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.cs
i.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_02_19.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://srtm.csi.
cgiar.org/wp-content/uploads/files/srtm_5x5/tiff/srtm_02_19.zip on http://srtm.c
si.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_02_19.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-c
si-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_19.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_19.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_19.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-c
si-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_20.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_20.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_20.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.cs
i.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_02_20.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://srtm.csi.
cgiar.org/wp-content/uploads/files/srtm_5x5/tiff/srtm_02_20.zip on http://srtm.c
si.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_02_20.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-c
si-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_20.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_20.zip on http://cgiar-csi
-srtm.openterrain.org.s3.amazonaws.com/source/srtm_02_20.zip
.10%...136129
90% done.
org.esa.snap.core.gpf.OperatorException: 136129
        at org.esa.snap.core.gpf.graph.GraphProcessor$GPFImagingListener.errorOc
curred(GraphProcessor.java:363)
        at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(SunTi
leScheduler.java:1646)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler
.java:921)
        at javax.media.jai.OpImage.getTile(OpImage.java:1129)
        at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(Operator
Context.java:407)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(Operator
Context.java:393)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:461)
        at org.esa.s3tbx.olci.radiometry.rayleigh.RayleighCorrectionOp.lambda$co
mputeTileStack$0(RayleighCorrectionOp.java:182)
        at java.util.HashMap$EntrySet.forEach(Unknown Source)
        at org.esa.s3tbx.olci.radiometry.rayleigh.RayleighCorrectionOp.computeTi
leStack(RayleighCorrectionOp.java:172)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(Ope
ratorImageTileStack.java:116)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(Ope
ratorImageTileStack.java:85)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler
.java:904)
        at javax.media.jai.OpImage.getTile(OpImage.java:1129)
        at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(Operator
Context.java:407)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(Operator
Context.java:393)
        at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImag
e.java:73)
        at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:
137)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler
.java:904)
        at javax.media.jai.OpImage.getTile(OpImage.java:1129)
        at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
        at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
Caused by: org.esa.snap.core.gpf.OperatorException: 136129
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorExceptio
n(OperatorUtils.java:415)
        at org.esa.snap.raster.gpf.masks.CreateLandMaskOp.computeTileStack(Creat
eLandMaskOp.java:259)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(Ope
ratorImageTileStack.java:116)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(Ope
ratorImageTileStack.java:85)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler
.java:904)
        ... 23 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 136129
        at org.esa.snap.core.datamodel.ProductData$Short.getElemDoubleAt(Product
Data.java:1523)
        at org.esa.snap.raster.gpf.masks.CreateLandMaskOp.computeTileStack(Creat
eLandMaskOp.java:227)
        ... 26 more

Error: 136129

Additional information:
I’m using Snap 8.0 (this might explain the srtm warnings?)

My question is: is there an alternative way of doing this, or a way to fix this error?

Thanks for the help,
Oli

marpet · October 20, 2020, 8:34am

Thanks for the report.

It seems that the SRTM tile 02-20 is not available. Maybe it is over water. However, the processing should go on.

it seems that the original problem occurs in the CreateLandMaskOp.
@lveci Can you have a look?

The rayleigh operator does not allow for specifying a valid-pixel expression.
But this could be a useful update.