GPT Subset loses some netCDF4 metadata

snap
#1

I apply the same .xml script using gpt to my netCDF4 files.
subset.xml (324 Bytes)

After running it on my home computer I get full NC_GLOBAL attributes set:

Driver: netCDF/Network Common Data Format
Files: 19980717_d-OC_PML-L3-CHL-occci_MULTI_4KM-GLO-REP-v02_Ntest.nc
Size is 4624, 2304
Coordinate System is `’
Metadata:
CHL#coordinates=lat lon
CHL#long_name=Chlorophyll-a concentration in seawater (not log-transformed), generated by as a blended combination of OCI, OCI2, OC2 and OCx algorithms, depending on water class memberships
CHL#units=milligram m-3
CHL#_FillValue=nan
NC_GLOBAL#citation=The licensees should respect the Copernicus Marine usage agreement (http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/) by crediting Copernicus in a manner similar to: <Generated using E.U. Copernicus Marine Service Information, provided by OCTAC/PML production centre>
NC_GLOBAL#cmems_production_unit=OC-PML-PLYMOUTH-UK
NC_GLOBAL#cmems_product_id=OCEANCOLOUR_GLO_CHL_L3_REP_OBSERVATIONS_009_065
NC_GLOBAL#comment=
NC_GLOBAL#contact=email: cmems@pml.ac.uk
NC_GLOBAL#Conventions=CF-1.4
NC_GLOBAL#creation_date=2020-10-27
NC_GLOBAL#Creation_time=12:00:00 UTC
NC_GLOBAL#distribution_statement=Copernicus Marine data license (see website)
NC_GLOBAL#file_quality_index=0
NC_GLOBAL#grid_mapping=grid_mapping
NC_GLOBAL#grid_resolution=4KM
NC_GLOBAL#history=Source data were: NASA OBPG SeaWiFS level1 R2018.0 LAC and GAC [A/C via l2gen], NASA OBPG VIIRS L1 R2018.0 [A/C via polymer], NASA OBPG MODIS Aqua L1 R2018.0 (A/C via polymer), ESA MERIS L1B (3rd reprocessing inc OCL correction) [Polymer v4.12] and OLCI L1B [Polymer v4.12]; Derived products were mainly produced with functions validated from the current NASA SeaDAS release and some custom implementations. Uncertainty generation determined by the fuzzy classifier scheme of Tim Moore (2009) as updated by Jackson et al. (2017) and Jackson et al. (in prep).
Remapped variable latitude_longitude to grid_mapping. Remapped variable chlor_a to CHL.
NC_GLOBAL#institution=Plymouth Marine Laboratory, operating as a production unit within the CMEMS OCTAC
NC_GLOBAL#keywords=satellite,observation,ocean,ocean colour
NC_GLOBAL#Metadata_Conventions=Unidata Dataset Discovery v1.0
NC_GLOBAL#metadata_profile=beam
NC_GLOBAL#metadata_version=0.5
NC_GLOBAL#Naming_authority=CMEMS
NC_GLOBAL#netcdf_file_type=NETCDF4_CLASSIC
NC_GLOBAL#Netcdf_version_id=V4
NC_GLOBAL#parameter=Chlorophyll-a concentration in seawater (not log-transformed), generated by as a blended combination of OCI, OCI2, OC2 and OCx algorithms, depending on water class memberships
NC_GLOBAL#parameter_code=CHL
NC_GLOBAL#platform=Orbview-2/SeaStar
NC_GLOBAL#product_level=3
NC_GLOBAL#product_type=CF-1.7
NC_GLOBAL#product_version=v5
NC_GLOBAL#project=Copernicus Marine (CMEMS)
NC_GLOBAL#references=http://marine.copernicus.eu
NC_GLOBAL#sensor=ESA Ocean Colour Climate Initiative v5
NC_GLOBAL#sensor_name=SeaWiFS
NC_GLOBAL#site_name=GLO
NC_GLOBAL#software_name=MERIS SeaDAS processor
NC_GLOBAL#software_version=7.5
NC_GLOBAL#start_date=17-JUL-1998 09:59:23.338735
NC_GLOBAL#start_time=00:00:00 UTC
NC_GLOBAL#stop_date=17-JUL-1998 09:59:23.338735
NC_GLOBAL#stop_time=23:59:00 UTC
NC_GLOBAL#summary=Data products generated by the Ocean Colour component of the European Space Agency Climate Change Initiative project. These files are daily composites of merged sensor (MERIS, MODIS Aqua, SeaWiFS LAC & GAC, VIIRS, OLCI) products. MODIS Aqua and SeaWiFS were band-shifted and bias-corrected to MERIS bands and values using a temporally and spatially varying scheme based on the overlap years of 2003-2007. VIIRS was band-shifted and bias-corrected in a second stage against the MODIS Rrs that had already been corrected to MERIS levels, for the overlap period 2012-2013; and at the third stage OLCI was bias corrected against already corrected MODIS, for overlap period 2016-07-01 to 2019-06-30. VIIRS, MODIS, SeaWiFS and MERIS Rrs were derived from a combination of NASA/s l2gen (for basic sensor geometry corrections, etc) and HYGEOS Polymer v4.12 (for atmospheric correction). OLCI Rrs were sourced at L1b (already geometrically corrected) and processed with polymer. The Rrs were binned to a sinusoidal 4km level-3 grid, and later to 4km geographic projection, by Brockmann Consult/s SNAP. Derived products were generally computed with the standard algorithmsthrough SeaDAS. QAA IOPs were derived using the standard SeaDAS algorithm but with a modified backscattering table to match that used in the bandshifting. The final chlorophyll is a combination of OCI, OCI2, OC2 and OCx, depending on the water class memberships. Uncertainty estimates were added using the fuzzy water classifier and uncertainty estimation algorithm of Tim Moore as documented in Jackson et al (2017). and updated accorsing to Jackson et al. (in prep).
NC_GLOBAL#TileSize=475:556
NC_GLOBAL#title=dataset-oc-glo-chl-multi_cci-l3-chl_4km_daily-rep-v02

However, when I tried to run it on an external computer, I received only:

Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Driver: netCDF/Network Common Data Format
Files: 19980701_d-OC_PML-L3-CHL-occci_MULTI_4KM-GLO-REP-v02_new.nc
Size is 8638, 2399
Origin = (-180.000000000000000,65.000000000000000)
Pixel Size = (0.041666666666657,-0.041666666666667)
Metadata:
CHL#coordinates=lat lon
CHL#long_name=Chlorophyll-a concentration in seawater (not log-transformed), generated by as a blended combination of OCI, OCI2, OC2 and OCx algorithms, depending on water class memberships
CHL#units=milligram m-3
CHL#_FillValue=-999
lat#long_name=latitude
lat#standard_name=latitude
lat#units=degrees_north
lat#valid_max=64.97916666666667
lat#valid_min=-34.9375
lon#long_name=longitude
lon#standard_name=longitude
lon#units=degrees_east
lon#valid_max=179.8958333332515
lon#valid_min=-179.9791666666667
NC_GLOBAL#Conventions=CF-1.4
NC_GLOBAL#start_date=01-JUL-1998 03:19:54.443158
NC_GLOBAL#stop_date=01-JUL-1998 16:38:52.234313
NC_GLOBAL#TileSize=540:576
NC_GLOBAL#title=dataset-oc-glo-chl-multi_cci-l3-chl_4km_daily-rep-v02

Are there any environmental variables that could impact the output? The same difference I obtain even when I add -PcopyMetadata==true . Could you please tell me where I may look for the reasons of the differences?

Regards,
Marta

#2

Have you checked that both systems are running the current SNAP 8.0.N with updates?

When you run gpt it looks for a suitable GDAL installation. I’m not sure if GPT uses GDAL to write NetCDF files, but the difference might result from one system using SNAP’s internal GDAL and the other using an external GDAL.

#3

Hi,

Yes, one thing is to check that the versions are equal as gnwiii said the other thing is to ensure that the same reader is used. Actually this should the same reader should be selected automatically, but maybe there is an issue. If 2 readers say they can read the file it is not deterministic which one is selected.
So you can add a read node to your graph.
Like:

<graph id="someGraphId">
    <version>1.0</version>
	<node id="readNode">
      <operator>Read</operator>
      <sources/>
      <parameters>
        <file>${source}</file>
        <formatName>NetCDF4-CF</formatName>
      </parameters>
    </node>
	
    <node id="cut_Npole">
      <operator>Subset</operator>
      <sources>
        <source>readNode</source>
      </sources>
      <parameters>
        <region>0,600,8638,2399</region>
        <copyMetadata>true</copyMetadata>
      </parameters>
    </node>
</graph>

Another thing. You are comparing two different files. Maybe they are different in metadata?
For example one says
CHL#_FillValue=nan
and the other
CHL#_FillValue=-999

You could check the metadata with ncdump or HDFView.

#4

Hi,

Thank you so much for help! I solved it.
It seems that it was not about Read node though, but the write node.

Anyway, I had to force the use of ‘NetCDF4-BEAM’ format (‘NetCDF4-CF’ shortened metadata)
by adding: -f ‘NetCDF4-BEAM’
while running processing, apart from specifying only the target filename:
-t {filename}

Thank you for help!

Marta