GeoTIFF-BigTIFF export creates empty file

jdries · January 31, 2020, 10:03am

I am exporting bands from a Sentinel-2 L1C product to geotiff, with some constraints:

direct export to geotiff (no BEAM-DIMAP!)
limit memory requirements as much as possible (e.g. 60m bands shouldn’t need much)

To achieve this, I’m playing around with all possible parameters that I’m aware of. This revealed a critical issue: some parameter combinations results in empty files (having 0 bytes), without showing errors.
My actual use case is more complex, but the example below shows the issue, using SNAP 7.0. Note that it is very sensitive to the memory and jai tile cache parameters. So by simply changing those, it can be made to work (assuming you feel like doing a lot of trial and error).

 /usr/local/snap/bin/gpt BandSelect -e -Dsnap.dataio.bigtiff.tiling.width=256 -Dsnap.dataio.bigtiff.tiling.height=256 -Dsnap.jai.defaultTileSize=256 -c 1000M  -q 4 -J-Xmx1000M -PsourceBands=B1 -f GeoTIFF-BigTIFF -t B01.tif S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826.zip

Example run

[driesj@driesj ~]$ time /usr/local/snap/bin/gpt BandSelect -e -Dsnap.dataio.bigtiff.tiling.width=256 -Dsnap.dataio.bigtiff.tiling.height=256 -Dsnap.jai.defaultTileSize=256 -c 1000M  -q 4 -J-Xmx1000M -PsourceBands=B1 -f GeoTIFF-BigTIFF -t B01.tif S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826.zip
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH does not contain the current folder '.'. Its value is '/opt/rh/python27/root/usr/lib64'.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32650
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.core.gpf.common.WriteOp: Start writing product S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826 to B01.tif
Writing.....
INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file B01.tif
..10%....20%....30%....40%....50%....60%....70%....80%....90%....100% done.
INFO: org.esa.snap.core.gpf.common.WriteOp: End writing product B01 to B01.tif
INFO: org.esa.snap.core.gpf.common.WriteOp: Time:  9.200 s total,  0.838 ms per line, 0.000076 ms per pixel

real    0m21.081s
user    0m44.087s
sys     0m5.674s

jdries · February 1, 2020, 11:01am

Some more relevant info, I got the GDAL based geotiff driver to work. This was non-trivial as the built-in attempts to load native libaries seem to fail. So I had to configure java.library.path myself.
It runs faster, and never resulted in an empty file. However, I cannot set gdal creation option like compression and tile size, so it does not solve my problem in the end, as I’m looking for ways to create a proper geotiff.

This is the full example run:

[driesj@driesj ~]$ LD_LIBRARY_PATH=. /usr/local/snap/bin/gpt BandSelect -e -J-Djava.library.path=/home/driesj/.snap/auxdata/gdal:/home/driesj/.snap/auxdata/gdal/gdal-2.2.0-linux/lib/jni:/home/driesj/.snap/auxdata/gdal/gdal-2.2.0-linux/lib/ -Dsnap.dataio.bigtiff.tiling.width=256 -Dsnap.dataio.bigtiff.tiling.height=256 -Dsnap.jai.defaultTileSize=256 -c 128M -x -q 4 -J-Xmx512M -PsourceBands=B1 -f GDAL-GTiff-WRITER -t B01.tif /data/TERRASCOPE/morpho_v2/unit_test_data/unit_test_data_develop/products/S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826/input/S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826.zip
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32650
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.core.gpf.common.WriteOp: Start writing product S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826 to B01.tif
Writing...
....10%....20%....30%....40%....50%.....62%....72%....82%....92%... done.
INFO: org.esa.snap.core.gpf.common.WriteOp: End writing product S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826 to B01.tif
INFO: org.esa.snap.core.gpf.common.WriteOp: Time:  4.007 s total,  0.365 ms per line, 0.000033 ms per pixel

jdries · February 1, 2020, 11:44am

There’s another bug in the gdal GTiff driver, my output products don’t seem to have proper georeferencing anymore:

Size is 5490, 5490
Coordinate System is:
GEOGCS["World Geodetic System 1984",
    DATUM["unknown",
        SPHEROID["unnamed",6378137,298.257223563]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","6326"]]
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 5490.0)
Upper Right ( 5490.0,    0.0)
Lower Right ( 5490.0, 5490.0)
Center      ( 2745.0, 2745.0)
Band 1 Block=5490x1 Type=Float32, ColorInterp=Gray

jdries · February 1, 2020, 1:43pm

@ramona_manda The bug that causes products written by GDALProductWriter to have incorrect georeferenceing can be found here:

github.com

senbox-org/s2tbx/blob/bc1cb18900f7c93fb5db029cf9e2b17df7613f52/s2tbx-gdal-reader/src/main/java/org/esa/s2tbx/dataio/gdal/writer/GDALProductWriter.java#L110


    this.gdalDataset = this.gdalDriver.Create(outputFile.toString(), imageWidth, imageHeight, bandCount, this.gdalDataType);
    if (this.gdalDataset == null) {
        throw new NullPointerException("Failed creating the file to export the product for driver '" + this.gdalDriver.getLongName() + "'.");
    }
    this.bandsMap = new HashMap<Band, org.gdal.gdal.Band>(bandCount);


    GeoCoding geoCoding = sourceProduct.getSceneGeoCoding();
    if (geoCoding == null) {
        this.gdalDataset.SetProjection("");
    } else {
        this.gdalDataset.SetProjection(geoCoding.getGeoCRS().toWKT());
    }
}


@Override
public void writeBandRasterData(Band sourceBand, int sourceOffsetX, int sourceOffsetY, int sourceWidth, int sourceHeight, ProductData sourceBuffer, ProgressMonitor pm) throws IOException {
    Guardian.assertNotNull("sourceBand", sourceBand);
    Guardian.assertNotNull("sourceBuffer", sourceBuffer);
    checkBufferSize(sourceWidth, sourceHeight, sourceBuffer);


    long sourceBandWidth = sourceBand.getRasterWidth();

Proposed fix:

this.gdalDataset.SetProjection(geoCoding.getMapCRS().toWKT());
org.geotools.referencing.operation.transform.AffineTransform2D transform = (org.geotools.referencing.operation.transform.AffineTransform2D) geoCoding.getImageToMapTransform();
double[] gdalGeoTransform = new double[6];
gdalGeoTransform[0] = transform.getTranslateX();
gdalGeoTransform[3] = transform.getTranslateY();
gdalGeoTransform[1] = transform.getScaleX();
gdalGeoTransform[5] = transform.getScaleY();
this.gdalDataset.SetGeoTransform(gdalGeoTransform);

I also have a proposal to allow creation options:

final String gdalWriteOptions = Config.instance().preferences().get("snap.dataio.gdal.creationoptions", "TILED=YES;COMPRESS=DEFLATE");
String[] options = StringUtils.stringToArray(gdalWriteOptions, ";");
this.gdalDataset = this.gdalDriver.Create(outputFile.toString(), imageWidth, imageHeight, bandCount, this.gdalDataType,options);

A fast fix would be appreciated, this is really a critical bug.
Pull request:

mdelgado · February 1, 2020, 3:59pm

Had you tried the simple solution proposed here to export individual Sentinel-2 jp2 bands into GTIFF here: Sentinel-2 individual jp2 bands to GeoTIFF

Good luck!

jdries · February 1, 2020, 5:39pm

No, my real use case is not about trying to convert individual bands to geotiff, I also want to apply other snap operations.
I just posted a minimal example that clearly exposes the issue(s), so that it’s easier to debug, and rule out influence from a more complex process graph.

jdries · February 5, 2020, 7:18am

Hi @marpet,
could you give us an idea on whether the proposed fixes could be integrated? We need to integrate this in a processing chain, so it would help a lot if we don’t need to maintain a patched version.
We can always improve the pull request if needed.

thanks!

simonevaccari · March 16, 2021, 4:27pm

Any updates on this? I am experiencing a similar problem but with Sentinel1 data: Graph execution failed with a Geotiff output created but empty after this log:

INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file S1A_IW_GRDH_1SDV_20201228T170552_20201228T170617_035889_0433FB_D8C7_SIGMA0_VV_DB.tif

This is the line that generates this error:

operator=Operator(“Write”, file=“output_fname”, formatName=“GeoTIFF-BigTIFF”)

Full error log message:

Processing the graph
Picked up _JAVA_OPTIONS: -Xms24g -Xmx24g
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
version = 3.31
INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file S1A_IW_GRDH_1SDV_20201228T170552_20201228T170617_035889_0433FB_D8C7_SIGMA0_VV_DB.tif
10%20%30%40%50%60%70%.80%90%
Traceback (most recent call last):
File “/srv/conda/envs/env_snap/bin/sar-calibration”, line 33, in
sys.exit(load_entry_point(‘sar-calibration==0.0.1’, ‘console_scripts’, ‘sar-calibration’)())
File “/srv/conda/envs/env_snap/lib/python3.9/site-packages/click/core.py”, line 829, in call
return self.main(*args, **kwargs)
File “/srv/conda/envs/env_snap/lib/python3.9/site-packages/click/core.py”, line 782, in main
rv = self.invoke(ctx)
File “/srv/conda/envs/env_snap/lib/python3.9/site-packages/click/core.py”, line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/srv/conda/envs/env_snap/lib/python3.9/site-packages/click/core.py”, line 610, in invoke
return callback(*args, **kwargs)
File “/srv/conda/envs/env_snap/lib/python3.9/site-packages/sar_calibration-0.0.1-py3.9.egg/sar_calibration/app.py”, line 33, in main
File “/srv/conda/envs/env_snap/lib/python3.9/site-packages/snapista-0.1.2-py3.9.egg/snapista/graph.py”, line 338, in run
Exception: Graph execution failed (exit code -9)

Thanks in advance! @mdelgado @jdries