My workflow is to use GPT to subset S3 OLCI products and then reproject them to the local UTM zone and 300 m resolution with something like
gpt reproject -Presampling=Bilinear -PpixelSizeX=300 -PpixelSizeY=300 -Pcrs=epsg:32719 -Ssource=<infile> -t<outfile>
On some scenes, this works fine. On others, however, my -Xmx32G
is not enough and I get this error:
.INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
.SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder '.'.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.core.gpf.common.WriteOp: Start writing product projected_S3A_OL_2_WFR____20180816T143016_20180816T143316_20180817T203805_0179_034_324_3240_MAR_O_NT_002 to /mnt/output/S3A_OL_2_WFR____20180816T143016_20180816T143316_20180817T203805_0179_034_324_3240_MA$
_O_NT_002.dim
Writing...
INFO: java.util.prefs.FileSystemPreferences$1: Created user preferences directory.
20%....30%....40%.... done.
Exception in thread "SunTileScheduler0Standard13" java.lang.NullPointerException
at com.sun.media.jai.util.SunCachedTile.<init>(SunCachedTile.java:80)
at com.sun.media.jai.util.SunTileCache.add(SunTileCache.java:257)
at javax.media.jai.OpImage.addTileToCache(OpImage.java:1087)
at javax.media.jai.OpImage.getTile(OpImage.java:1142)
at org.esa.snap.core.gpf.internal.OperatorExecutor$OperatorTileComputationListenerStack.tileComputed(OperatorExecutor.java:310)
at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:278)
at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
org.esa.snap.core.gpf.OperatorException: Cannot construct DataBuffer.
at org.esa.snap.core.gpf.internal.OperatorExecutor$GPFImagingListener.errorOccurred(OperatorExecutor.java:376)
at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(SunTileScheduler.java:1646)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:921)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:407)
at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:393)
at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:73)
at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
Caused by: java.lang.RuntimeException: Cannot construct DataBuffer.
at com.sun.media.jai.util.DataBufferUtils.constructDataBuffer(DataBufferUtils.java:132)
at com.sun.media.jai.util.DataBufferUtils.createDataBufferFloat(DataBufferUtils.java:214)
at javax.media.jai.ComponentSampleModelJAI.createDataBuffer(ComponentSampleModelJAI.java:271)
at javax.media.jai.RasterFactory.createWritableRaster(RasterFactory.java:691)
at javax.media.jai.PlanarImage.createWritableRaster(PlanarImage.java:1982)
at javax.media.jai.PointOpImage.computeTile(PointOpImage.java:771)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
... 11 more
Error: Cannot construct DataBuffer.
When I then set -Xmx128G
, the RAM usage actually goes up to that staggering number and the job succeeds. A product that fails in this way is S3A_OL_2_WFR____20180816T143016_20180816T143316_20180817T203805_0179_034_324_3240_MAR_O_NT_002
Could be related to the issue of reproject creating ghost pixels and the image becomes huge - although I have never seem those pixels in the final output.
How can this task consume so much memory? And how can I fix this? I cannot allocate 128 GB of memory for every reprojection job…
By the way, 20%....30%....40%.... done.
- is that output from GDAL running behind the scenes?