GeoTIFF-BigTIFF export creates empty file

I am exporting bands from a Sentinel-2 L1C product to geotiff, with some constraints:

  • direct export to geotiff (no BEAM-DIMAP!)
  • limit memory requirements as much as possible (e.g. 60m bands shouldn’t need much)

To achieve this, I’m playing around with all possible parameters that I’m aware of. This revealed a critical issue: some parameter combinations results in empty files (having 0 bytes), without showing errors.
My actual use case is more complex, but the example below shows the issue, using SNAP 7.0. Note that it is very sensitive to the memory and jai tile cache parameters. So by simply changing those, it can be made to work (assuming you feel like doing a lot of trial and error).

 /usr/local/snap/bin/gpt BandSelect -e -Dsnap.dataio.bigtiff.tiling.width=256 -Dsnap.dataio.bigtiff.tiling.height=256 -Dsnap.jai.defaultTileSize=256 -c 1000M  -q 4 -J-Xmx1000M -PsourceBands=B1 -f GeoTIFF-BigTIFF -t B01.tif S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826.zip

Example run

[driesj@driesj ~]$ time /usr/local/snap/bin/gpt BandSelect -e -Dsnap.dataio.bigtiff.tiling.width=256 -Dsnap.dataio.bigtiff.tiling.height=256 -Dsnap.jai.defaultTileSize=256 -c 1000M  -q 4 -J-Xmx1000M -PsourceBands=B1 -f GeoTIFF-BigTIFF -t B01.tif S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826.zip
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH does not contain the current folder '.'. Its value is '/opt/rh/python27/root/usr/lib64'.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32650
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.core.gpf.common.WriteOp: Start writing product S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826 to B01.tif
Writing.....
INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file B01.tif
..10%....20%....30%....40%....50%....60%....70%....80%....90%....100% done.
INFO: org.esa.snap.core.gpf.common.WriteOp: End writing product B01 to B01.tif
INFO: org.esa.snap.core.gpf.common.WriteOp: Time:  9.200 s total,  0.838 ms per line, 0.000076 ms per pixel

real    0m21.081s
user    0m44.087s
sys     0m5.674s

Some more relevant info, I got the GDAL based geotiff driver to work. This was non-trivial as the built-in attempts to load native libaries seem to fail. So I had to configure java.library.path myself.
It runs faster, and never resulted in an empty file. However, I cannot set gdal creation option like compression and tile size, so it does not solve my problem in the end, as I’m looking for ways to create a proper geotiff.

This is the full example run:

[driesj@driesj ~]$ LD_LIBRARY_PATH=. /usr/local/snap/bin/gpt BandSelect -e -J-Djava.library.path=/home/driesj/.snap/auxdata/gdal:/home/driesj/.snap/auxdata/gdal/gdal-2.2.0-linux/lib/jni:/home/driesj/.snap/auxdata/gdal/gdal-2.2.0-linux/lib/ -Dsnap.dataio.bigtiff.tiling.width=256 -Dsnap.dataio.bigtiff.tiling.height=256 -Dsnap.jai.defaultTileSize=256 -c 128M -x -q 4 -J-Xmx512M -PsourceBands=B1 -f GDAL-GTiff-WRITER -t B01.tif /data/TERRASCOPE/morpho_v2/unit_test_data/unit_test_data_develop/products/S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826/input/S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826.zip
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32650
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.core.gpf.common.WriteOp: Start writing product S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826 to B01.tif
Writing...
....10%....20%....30%....40%....50%.....62%....72%....82%....92%... done.
INFO: org.esa.snap.core.gpf.common.WriteOp: End writing product S2B_MSIL1C_20180418T030539_N0206_R075_T50TMK_20180418T055826 to B01.tif
INFO: org.esa.snap.core.gpf.common.WriteOp: Time:  4.007 s total,  0.365 ms per line, 0.000033 ms per pixel

There’s another bug in the gdal GTiff driver, my output products don’t seem to have proper georeferencing anymore:

Size is 5490, 5490
Coordinate System is:
GEOGCS["World Geodetic System 1984",
    DATUM["unknown",
        SPHEROID["unnamed",6378137,298.257223563]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","6326"]]
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 5490.0)
Upper Right ( 5490.0,    0.0)
Lower Right ( 5490.0, 5490.0)
Center      ( 2745.0, 2745.0)
Band 1 Block=5490x1 Type=Float32, ColorInterp=Gray

@ramona_manda The bug that causes products written by GDALProductWriter to have incorrect georeferenceing can be found here:

Proposed fix:

this.gdalDataset.SetProjection(geoCoding.getMapCRS().toWKT());
org.geotools.referencing.operation.transform.AffineTransform2D transform = (org.geotools.referencing.operation.transform.AffineTransform2D) geoCoding.getImageToMapTransform();
double[] gdalGeoTransform = new double[6];
gdalGeoTransform[0] = transform.getTranslateX();
gdalGeoTransform[3] = transform.getTranslateY();
gdalGeoTransform[1] = transform.getScaleX();
gdalGeoTransform[5] = transform.getScaleY();
this.gdalDataset.SetGeoTransform(gdalGeoTransform);

I also have a proposal to allow creation options:

final String gdalWriteOptions = Config.instance().preferences().get("snap.dataio.gdal.creationoptions", "TILED=YES;COMPRESS=DEFLATE");
String[] options = StringUtils.stringToArray(gdalWriteOptions, ";");
this.gdalDataset = this.gdalDriver.Create(outputFile.toString(), imageWidth, imageHeight, bandCount, this.gdalDataType,options);

A fast fix would be appreciated, this is really a critical bug.
Pull request:

1 Like

Had you tried the simple solution proposed here to export individual Sentinel-2 jp2 bands into GTIFF here: Sentinel-2 individual jp2 bands to GeoTIFF

Good luck!

No, my real use case is not about trying to convert individual bands to geotiff, I also want to apply other snap operations.
I just posted a minimal example that clearly exposes the issue(s), so that it’s easier to debug, and rule out influence from a more complex process graph.

1 Like

Hi @marpet,
could you give us an idea on whether the proposed fixes could be integrated? We need to integrate this in a processing chain, so it would help a lot if we don’t need to maintain a patched version.
We can always improve the pull request if needed.

thanks!