Problem saving product as HDF5

development python
#1

Hello all,
I am using python and snappy to copy data from a source product (geo-tiff) to a larger target product and I encountered the following problems:

  • When saving the target product as HDF5, no geolocation is saved in the product, i.e. when opening it in SNAP, the pixels only have the value, no latitude/longitude.
  • When making the target product larger (10.000x10.000 pixels) the method “target_band.ensureRasterData()” gives a OutOfMemoryError.

Here the relevant part of my code:

source = snappy.ProductIO.readProduct(path)
b = source.getBand(param)

target = snappy.Product(source.getProductType(), source.getProductType(), 10000, 10000)
snappy.ProductUtils.copyProductNodes(source, target)

type = snappy.ProductData.TYPE_INT32
target_band = snappy.Band('_world', type, 10000, 10000)
snappy.ProductUtils.copyRasterDataNodeProperties(b, target_band)
target.addBand(target_band)

target_band.setNoDataValue(np.nan)
target_band.setNoDataValueUsed(True)
target.setProductWriter(snappy.ProductIO.getProductWriter('HDF5'))
target.writeHeader(snappy.String('target.h5'))
target_band.ensureRasterData()
gc_target = target.getSceneGeoCoding()
map_crs = gc_target.getMapCRS()
gc_target = CrsGeoCoding(map_crs, 10000,10000,7.0,48.0,0.0027,0.0027)


#set pixel values with "target_band.setPixel(...)"

snappy.ProductIO.writeProduct(target, 'target.h5', 'HDF5')
target.closeIO()

Am I using the ProductWriter wrongly? Is there another way to ensure that the RasterData and the GeoCoding is set correctly?
Thanks for any help!

#2

HDF5 may be too low-level. NetCDF4-CF combines (a restricted subset) HDF5 with the CF meta-data conventions so there is support for geolocation of grids using lon,lat vars. I’m not sure about current CF versions, but older versions lacked support for CRS data, so most apps rely on lon,lat vars.

#3

Thanks for that information. This works a lot better than HDF5 :slight_smile:

However, I still get an OutOfMemory Error for
target_band.ensureRasterData()
And if I do not use that method I get NullPointerException when trying to set the pixels.

Is there any other way to set this?
Thanks!

#4

Hi Helga,
In the provided code your are only getting and change the GeoCoding. But you also need to set the changed value to the target product. Try

target.setSceneGeoCoding(gc_target)

But probably you just want to copy from the source product. In this case try

snappy.ProductUtils.copyGeoCoding(source, target)

The problem with target_band.ensureRasterData() indicates two things the JVM started by your snappy does not have a lot memory assigned. See:

You need to change the configuration in the snappy.ini file. The file is located in the snappy folder.
Change the line
# java_max_mem: 4G
to e.g.
java_max_mem: 6G

This means that snappy can use 6GB of your RAM. A recommended value is 70%-80% of the available RAM in your system.

If this does not help you can iterate over the dimensions of the band.
target_band.ensureRasterData() means that the whole band is loaded at once, so a lot of memory is allocated.

Have a look at the followingexample. Here the data is processed line-by-line. I guess you can do it in a similar way.

From tif to product