GPT processing of Sentinel-1 produces invalid NetCDF image

[nena.v]

We have a Sentinel-1 GPT processing workflow which applies the following steps to each raw image:

• Read
• Apply-Orbit-File
• ThermalNoiseRemoval
• Calibration
• Terrain-Correction
• Subset
• LinearToFromdB
• Write [to NetCDF4-BEAM]

GPT Graph: sentinel1_gpt_EU01.xml (5.1 KB)

This workflow almost always works and produces a valid NetCDF. However in seemingly random cases, it produces an invalid NetCDF that cannot be read by other NetCDF readers. In another software I get errors like this: “There was an error opening this dataset at 4775020970896772037 file length = 4775020970896772037 Unexpected end of file”

An example of an image that fails is:
S1A_IW_GRDH_1SDV_20211220T231702_20211220T231730_041099_04E1F9_46EE.zip

When I load the raw Sentinel-1 image into the Desktop application, it shows data inside the subset area. I manually rebuilt the GPT processing steps in the SNAP Desktop application and it also produced an invalid NetCDF4-BEAM file for this Sentinel-1 image. Strangely, when I change the format to NetCDF4-CF, it produces a valid file.

Questions: Why does this fail for some Sentinel-1 images? Will changing the export format to NetCDF4-CF produce a more robust workflow? I recall having issues with NetCDF4-CF a year or two ago, which is why we ended up using NetCDF4-BEAM in the first place.

Note: I also run workflows for Sentinel-2 (producing NetCDF4-BEAM files) and RCM (producing NetCDF4-CF files), and this only seems to be a problem for Sentinel-1 imagery.
Using SNAP 8.0.9.

[gnwiii]

@nena.v

4775020970896772037

is a very big number, so likely the fictional creation of some bug in the reader.

You may have seen:

@marpet

The NetCDF-BEAM stores some BEAM/SNAP specific stuff, like colour information, expressions etc.
The data can be read in other applications too, but some information might be lost.

The NetCDF-CF is of more general use. The data is converted in a way that other applications can read it. This is the suggested format when you want to distribute the data. More specifically NetCDF4-CF, because this supports compression.

Support for NetCDF4-CF has improved in recent years as more people use it and find issues. If you still encounter problems they should be reported to the upstream developers.

Normally, applications ignore metadata they don’t use. There have been issues with applications that target smaller files and can’t handle large data sets.

NetCDF4 is a subset of HDF5. It would be interesting to know
if the “Unexpected end of file” error occurs when you load it as
HDF5 in applications widely used with HDF5/NetCDF4 data (HDFview, Matlab, IDL, etc).

It may be helpful to check how various widely tools interpret NetCDF4 metadata (e.g., using ncdump -h, gdalinfo, cdo info) for the two files. All these have large user communities so someone may recognize your issue.