Problems performing BRR correction after subset or band extraction in GPT

Optical Toolbox
#1

Hi Step Forum,

I have a rather complicated graph to run involving BRR correction and IdePix flagging of OLCI L1B EFR and ERR products. While I can run it on my laptop (just! - more on a request for help with TileCache in a parallel ticket), I am working on improving the run efficiency. First thing on my list was to implement band subsetting directly after ingestion to drop the global cache requirements. However, I have found that I am unable to perform either the GPT subset or band-extraction operations prior to the Rayleigh corrector. The error is as below (same for both tests):

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.2.1 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
90% done.
org.esa.snap.core.gpf.OperatorException
    at org.esa.snap.core.gpf.graph.GraphProcessor$GPFImagingListener.errorOccurred(GraphProcessor.java:379)
    at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(SunTileScheduler.java:1646)
    at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:921)
    at javax.media.jai.OpImage.getTile(OpImage.java:1129)
    at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
    at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
    at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
    at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
    at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:116)
    at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
    at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
    at javax.media.jai.OpImage.getTile(OpImage.java:1129)
    at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
    at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
Caused by: java.lang.NullPointerException
    at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:440)
    at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
    at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
    at org.esa.s3tbx.olci.radiometry.rayleigh.RayleighCorrectionOp.waterVaporCorrection709(RayleighCorrectionOp.java:321)
    at org.esa.s3tbx.olci.radiometry.rayleigh.RayleighCorrectionOp.lambda$computeTileStack$0(RayleighCorrectionOp.java:253)
    at java.util.HashMap$EntrySet.forEach(HashMap.java:1044)
    at org.esa.s3tbx.olci.radiometry.rayleigh.RayleighCorrectionOp.computeTileStack(RayleighCorrectionOp.java:211)
    at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
    at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
    at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
    ... 11 more

Running the same graph through the GUI Graph Builder just gives a null point error. However, I can run these process without issue if I step through them in the GUI. The bands I subset work in the GUI. I am copying the metadata and tie point grids (although I may not need to). I have removed the problematic <referenceBand>string</referenceBand> parameter from subset. I don’t think my GPT settings matter but in case they do, I am running with “-c 8027M” with gpt.vmoptions as:

-Xmx15G
-Xms8G
-XX:+AggressiveOpts
-Xverify:none
-Dsnap.dataio.netcdf.metadataElementLimit=10000
-Dsun.java2d.noddraw=true
-Dsun.awt.nopixfmt=true
-Dsun.java2d.dpiaware=false

Removing everything from -XX:+AggressiveOpts onwards does not help. The test graphs are attached;
BRR_bandex.xml (1.9 KB)
BRR_subset.xml (1.9 KB)

I am running with Snap 8.0 (Desktop and engine 8.0.8) with a fully updated S3Tbx, on OSx Big Sur.

Is this a known issue? Any help would be greatly appreciated as always.

Rad2Refl after Rayleigh correction
#2

I’m not sure what the BandsExtractorOp causes issues here, but in general I would suggest to use the Subset operator.
In addition, limiting the product to certain bands will not help to save memory in this case. The ReayleighOperator will only read the data which is needed for the result. Actually adding an extra operator can be contradictory. Another instance of the modified is needed to be kept in memory.

And because of the pull-processing model gpt is using it reads only the data into memory which is necessary for the output. For example it is not important when a regional subset is done. It can be done at the beginning of a graph or at the end. Only the necessary parts are computed.
Resampling and reprojection are operations where the order of operation make a difference and of course operatiosn where the actual values are changed.

The two graph files should work. One is with subset at the beginning the other not. I would use the one without explicit subsetting.

1 Like
#3

Thanks @marpet. My thinking was that a band subset would cause the rest of the band data to be “dumped” into garbage collection, but from your description it is only read-on-demand - which makes more sense. Also, I was able to run your example without a problem…leading me to question why mine would not run. I looked at the differences between BRR_subset.xml and Olci_Rayleigh_Subset.xml. Other than a slightly different ingestion method (which I also tested) the main disparity was in the bands being corrected. I systematically added them back in…and the addition of Oa11_radiance causes both BRR_subset.xml and Olci_Rayleigh_Subset.xml to fail. Oa10_radiance is fine. Oa12_radiance is fine. I tried it on 3 OLCI EFR products…same failed result. It fails even if you run with Oa11_radiance on its own (so probably not memory related). There is no issue at all running with Oa11_radiance if you don’t subset. Curious to know if you see the same. I have not tested if the band-extractor suffers the same.

I solved my general issue, however, finding it far more efficient to split my graph into two (BRR and IdePix in one half, and everything else in the latter), so this is a bit of a moot point now. Still, not sure why one single specific band is causing problems.

#4

Thanks Ben, you found an issue in the processor.
It is actually an issue of validating the input properly. If a band at wavelength ~709 shall be corrected also bands at 885 and 900 need to be available to derive the correction factors for the 709. Those are the bands Oa18_radiance and Oa19_radiance for OLCI.
It is not necessary to add them to the bands which are corrected but the need to be present in the source product.
I’ve updated the validation of the input product. With the next update a proper error message will appear.

[SIIITBX-401] Rayleigh Correction fails if band Oa11_radiance is included for correction - JIRA (atlassian.net)

Thanks, for the report.

1 Like