From tif to product

development python
#1

Hello everyone,

I’m trying to transform a tif file with one Sentinel 2’s band, to a product.
From what I read on other topics, I barely succeed, but when I’m writing this product to a dim file, I get a lot of warnings/errors.
The same goes if I’m writing another product to a dim file, but only if I transformed a tif to a dim before.
I suppose then that I’m changing a parameter that shouldn’t be changed, or not in this way.
Here is the part of the code I use to transform from tif to dim:

	image=gdal.Open(B1.tif)
	image_array=image.ReadAsArray()

	height = image_array.shape[0]
	width = image_array.shape[1]

	product = snappy.Product('title', 'subtitle', width, height)
	band_ = product.addBand('B1', snappy.ProductData.TYPE_FLOAT32)

	ProductUtils.copyGeoCoding(original_product, product)
	ProductUtils.copyProductNodes(original_product, product)
	ProductUtils.copyMetadata(original_product, product)

	writer = snappy.ProductIO.getProductWriter('BEAM-DIMAP')
	product.setProductWriter(writer)
	product.writeHeader('B1.dim')
	band_.writePixels(0, 0, image.RasterXSize, image.RasterYSize, image_array)
	
	ProductIO.writeProduct(product, "B1", 'BEAM-DIMAP')

P.S.: original_product is the product read from the raw S2 data.

From my point of view, it’s a problem with ProductUtils or writer/header.

Thank you in advance!

#2

The (Geo)TIFF format does not provide a standard way to store the full metadata needed for remote sensing applications. Some applications do store metadata in xml form, either embedded in the TIFF file or as an external file. Unless such a mechansim was used in writign the TIFF file, much of the metadata will be lost. Even if the full metadata are stored, gdal may ignore it. It might help to tell us how the TIFF files are created and show us the error messages you get. There are programs that will display the structure of TIFF files and may show if additional metadata are present.

#3

Thank you very much for your answer!
The tif file I’m trying to transform as a product is the result of an upsampling with a machine learning process.
When I’m extracting the band of my interest, can I save the metadata link to it somewhere?
Or can I use the metadata of another band of the same raw product, copy it and link it to the new tif file generated from the band?
Finally, which programs are you talking about to display the metadata of a tif file?

Here are the errors/warnings:

JAI error occurred: 'Problem occurs when computing a tile by the owner.'
at com.sun.media.jai.util.SunTileScheduler@66107a2d
java.lang.IllegalStateException: no product reader for band 'B1'

In SNAP Desktop, when I’m trying to open the band:

DimapProductReader: Unable to read file 'my_path/B1.img' referenced as 'B1'

#4

This error can occur when you move the BEAM-DIMAP file to another location but you forget to move the corresponding data directory.
A BEAM-DIMAP consists of a *.dim file and a *.data directory.

Regarding thiw writing issue.
You mix to different writing scenarios. Yes, this API can be confusing, I know.

You can either replace the writePixels by setPixels call. In this case you don’t need get and set the writer before.

The other option is to remove the ProductIO.writeProduct() and call instead
writer.closeIO()

check also this post:
Problem saving product as HDF5 - development / python - STEP Forum (esa.int)
and especially the link to the ndvi example on github.
There are also a few more examples:
examples at master on GitHub

#5

Thank you for giving me more information.
I tested before with setPixels:

image=gdal.Open(B1.tif)
image_array=image.ReadAsArray()

height = image_array.shape[0]
width = image_array.shape[1]

product = snappy.Product('title', 'subtitle', width, height)
band_ = product.addBand('B1', snappy.ProductData.TYPE_FLOAT32)

ProductUtils.copyGeoCoding(original_product, product)
ProductUtils.copyProductNodes(original_product, product)
ProductUtils.copyMetadata(original_product, product)

#writer = snappy.ProductIO.getProductWriter('BEAM-DIMAP')
#product.setProductWriter(writer)
#product.writeHeader('B1.dim')
#band_.writePixels(0, 0, image.RasterXSize, image.RasterYSize, image_array)

band_.setPixels(0, 0, image.RasterXSize, image.RasterYSize, image_array)

ProductIO.writeProduct(product, "B1", 'BEAM-DIMAP')

but like some others, I’m getting the error:

RuntimeError: java.lang.NullPointerException

About the other option, if I just remove ProductIO.writeProduct() and call writer.closeIO():

#ProductIO.writeProduct(product, "B1", 'BEAM-DIMAP')
writer.closeIO()

then I get:

AttributeError: 'org.esa.snap.core.dataio.ProductWriter' object has no attribute 'closeIO'

#6

Oh, yes.

closeIO() is a method of Product.
For the Writer it is just close()

#7

You may can combine data from the machine learning produced tif file (call it mltif) with metadata of an existing product. I’d recomment using BEAM-DIMAP as metadata is in ASCII files. You should create a new band with the resolution of the mltif, then you can use the following process with ESA SNAP snappy

beampy_addnav.py (1.8 KB) (originally for use on Windows with ESA BEAM and NetCDF)

If you are doing many files you might do this more efficently with BEAM-DIMAP by editing the .dim and .hdr files, using the snappy script to create a template.

1 Like
#8

Basically, it’s what I’m already doing. If you take a look at my script above, I’m copying Geocoding of my original product, to the product generated from the MLtif. The errors/warnings are still there, unfortunately.

Then I tried what you told me Marpet, and it seems to work. I have to check it more deeply, but when I’m writing the product aften reading it from the MLtif, I don’t have issues raised anymore.
Thank you very much to both of you!

#9

After checking, if I want to work on a product opened before this “tif to product”, it doesn’t work with still the same issue.

Do you have some clues about what is happening? I still think that I change some parameters during the “tif to product” which I have to reset after to not alter the way of working on other products.

#10

Glad it is working Note that there is often important metadata other than geocoding that gets discarded or obscured in GeoTIFF files, so if you encounter issues with metadata then try BEAM-DIMAP.

#11

Once again, I need a bit of help.
With the @marpet’s solution, I can have my product with the “MLtif”. I have to write it and then read it to fully use it, but it’s okay (even if I still have some errors/warnings).

When I want to collocate it to another product, both are over the same AOI etc, but there is a problem of scale or something. I’m thinking about the pixel spacing still set to 60m instead of 10m. How can I take the geotransform from the second product and put it to the MLtif product? To replace it.
Thank you in advance!

#12

The errors/warnings may provide clues. I often see warnings about unknown TIFF tags when reading images from 3rd party applications. Some applications store metadata in serialized ASCII (xml, json, etc.), e.g. OME-TIFF.

You could try comparing metadata for the two products using gdalinfo and possibly construct a gdal VRT file with corrected metadata (VRT files are commonly used to supply geolocation for images, like PNG, that lack geolocation metadata). I’m not sure that a VRT will override faulty geolocation in a GeoTIFF file, but it is worth trying. Once you understand where the error occurs you may be able to use a GDAL tool to write a correct file. See: what is the best Pythonic way to save image metadata alongside a tif?