Write to NetCDF4-CF combined with subset produces invalid NetCDF file

snap
#1

We have a GPT script which is processing “S1B_IW_GRDH_1SDV_*” files and writing to NetCDF4-CF. The .nc files produced can be read into other software packages compatible with NetCDF.

However, when we add a “Subset” step to GPT just before the “Write” step, the NetCDF produced is no longer a valid NetCDF and cannot be opened, neither in SNAP nor in other NetCDF readers. SNAP gives the error “No appropriate product reader found. File XX can not be opened.”

We are subsetting using Geographic Coordinates and a rectangle defined using:
POLYGON ((-111.436 56.725, -111.4 56.725, -111.4 56.713, -111.436 56.713, -111.436 56.725))

The Subset step appears to be working. When we save to a different file format (BEAM-DIMAP), it can be opened by SNAP without any issues.

How can we subset our dataset and also produce a valid NetCDF file?

Update: I also tried using NetCDF4-BEAM. The NetCDF can be read into a NetCDF visualization software, and I can view the band information. However, the lat lon coordinates don’t appear to have been subset (they have the original lat/lon), and the dimensions of the band are called y1 and x1 instead of lat lon or x y…

Update 2: I should also mention that we’ve previously used Sentinel-3A products which were subset and exported to NetCDF4-BEAM (using snappy rather than GPT) and had no issues, so it seems to be either a Sentinel-1 or GPT-related issue.

GPT without full SNAP installation?
#2

It turns out that we were somehow still using SNAP version 6.0, though I recently did the installation. Once we updated to SNAP 7.0.3 this issue was resolved (the NetCDF4-BEAM latitude/lat and longitude/lon were consistent with each other, for the Sentinel-1 images).

#3

Hi everybody!
Unfortunately I have the same Issue with SNAP 7.0.3 and Sentinel-2 data.

  • I load a scene
  • I subset it manually by geocoordinates
  • I save it as netCDF 4-CF
  • I retrieve an erroneous *.nc file without lat/lon information and I guess therefore also wrong cell size.

I have tried with L1C and L2A data with no difference.
reading the .nc in R{raster} package also doesn´t seem to find the right structure in the metadata.
Conversion of the complete scene to NetCDF seems to output the CRS but aswell the wrong resoulution of 11 (should be 10*10).

Any help for outputting a correctly formatted netCDF CF-4 from SNAP after subsetting S2 scenes would be greatly appreciated.
Thanks,

Konrad

Does anybod

#4

Hi Konrad,
I also ran into some issues when I tried to subset my Sentinel-2 data. I think it has to do with the Sentinel-2 coordinates. I was able to solve it by adding a “Reproject” step before subsetting, to make sure the Sentinel-2 data were in geographic coordinates (rather than projected). I reprojected it to WGS84(DD).

I should add that I also had several issues with NetCDF4-CF in general, and ended up swapping to NetCDF4-BEAM, which I’m able to read into some other software packages.
Hope that helps!

1 Like
#6

Unfortunatly, only WGS84 is supported for NetCDF-CF. Other CRS are not, like UTM.
That is the underlying problem for your issue. And actually it is a shame that we don’t support in SNAP.

Netcdf export with x/y posting
#7

@ Nena,

thanks a lot for sharing your solution!
I reprojected the initial S2 image to WGS84(DD) with geographic coordinates (it ships with UTM), and did resampling and subsetting afterwards resulting in NetCDF-files with specified coordinate system in the metadata.
However if I reopen the initial S2-Scene zip file, and the NetCDF-BEAM file, the “zoom” for both images is not the same. Coordinates seem to be the same for each pixel. But the reprojected ones seem a bit stretched and also appear to be “deeper in the globe”.

Which other packages are you working with?
Because with the resulting NetCDFs i still cannot load multiple layers in R as it should be possible with the “brick” function from the R package raster.

@marpet thank you for sharing the issue. Is it the same for NetCDF-BEAM? It is true that it should be supported. I wasn´t expecting this kind of error in an early stage of my work. And in fact EO data from satellites is often compared with climate variable measurements, so putting this data together in a CF-Convention file seems to be a logical step.

Cheers and thank you so far,
Konrad

#8

I’ve been able to open the subsetted/projected/export-to-NetCDF data using Panoply (free software for viewing NetCDF files). Does R require a constant grid cell size? Because I’m noticing that when projecting to WGS84, the grid cell size in the exported NetCDF is not always constant (tiny differences). However, Panoply doesn’t seem to mind this… To figure this out, I opened the NetCDF in Panoply, exported the lat/lon coordinates to CSV, and viewed them in Excel.

I’m assuming that if UTM was supported in the NetCDF exported from SNAP, then the grid cell size would be consistent (e.g. 10 m for B2, B3, B4 bands). That would be idea… but from @marpet’s response, I understand this is not possible.

#9

Yes, it’s the same issue for NetCDF-BEAM

#10

Hi @marpet,
do you plan to solve this issue? So that NetCDF-CF output also supports projected formats? It would be really helpful to produce data with meaningful metadata in the widely accepted NetCDF format with CF convention.
Best,
Peter

#11

Hi Peter,

Yes, this is planned, but not very soon. The demand wasn’t very high in the past for this feature.
Currently we still working in the SNAP 8 release, and afterwords we will go throught the requirements for the next release and intermediate updates. So it might be solved with some update after SNAP 8.

#12

I tried this solution with SNAP 8 and during the write of the product either with NetCDF4-CF format and NetCDF4-BEAM but i recive this error:

Warning! ***HDF5 library version mismatched error
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as ‘LD_LIBRARY_PATH’.
You can, at your own risk, disable this warning by setting the environment
variable ‘HDF5_DISABLE_VERSION_CHECK’ to a value of ‘1’.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.10.3, library is 1.10.6

And i use this code:
parameters_export = snappy.HashMap()

parameters_export.put('clearCacheAfterRowWrite','false')

parameters_export.put('deleteOutputOnFailure','true')

parameters_export.put('file', output_name_file_noformat)

parameters_export.put('formatName','NetCDF4-BEAM') # NetCDF4-CF

parameters_export.put('writeEntireTileRows','false')

export_product = snappy.GPF.createProduct('Write', parameters_export, file_nc)

Thanks