Hi,
When we upgraded to SNAP 8, the output type of our processing changed from int to double. The operations we run are just Reproject and Subset. Is this expected?
Thanks,
Simon
Probably, yes.
I guess the output data format you are using is GeoTiff, right? As input, you use Sentinel-3, OLCI or SLSTR?
Have you changed the geo-coding setting for the data?
If you have switched to pixel-based geo-coding then the latitude/longitude bands are copied. These bands are of type double. You can exclude them in sour subset step.
If this is not the case, then let us know the data you are using, the output format and also the settings of your processing steps.
Just OLCI with geotiffs. We haven’t changed geocoding, and subsetting just the data bands does not help.
I just did the processing and the output type is float32.
Here shown in QGIS:
I have used this graph.
myGraph.xml (3.0 KB)
Now I notice that you say the data type was int before. I wonder how this could be. If you ould have used the spectral bands it should have always be float32. Only the quality_flags, detector_index and frame_offset bands could be written as int.
Here is the “graph.xml” that we use for Sentinel-3 processing.
<graph id="Graph">
<version>1.0</version>
<node id="Read">
<operator>Read</operator>
<sources/>
<parameters class="com.bc.ceres.binding.dom.XppDomElement">
<file>/tmp/tmpo3kir6pz_input/S3A_OL_1_EFR____20210301T214636_20210301T214936_20210301T233636_0179_069_086_2700_LN1_O_NR_002.SEN3/xfdumanifest.xml</file>
<formatName>Sen3</formatName>
</parameters>
</node>
<node id="Reproject">
<operator>Reproject</operator>
<sources>
<sourceProduct refid="Read"/>
</sources>
<parameters class="com.bc.ceres.binding.dom.XppDomElement">
<wktFile/>
<crs>AUTO:42001</crs>
<resampling>Bicubic</resampling>
<orthorectify>false</orthorectify>
<elevationModelName/>
<noDataValue>NaN</noDataValue>
<includeTiePointGrids>true</includeTiePointGrids>
<addDeltaBands>false</addDeltaBands>
</parameters>
</node>
<node id="Convert-Datatype">
<operator>Convert-Datatype</operator>
<sources>
<sourceProduct refid="Reproject"/>
</sources>
<parameters class="com.bc.ceres.binding.dom.XppDomElement">
<sourceBands>Oa01_radiance,Oa02_radiance,Oa03_radiance,Oa04_radiance,Oa05_radiance,Oa06_radiance,Oa07_radiance,Oa08_radiance,Oa09_radiance</sourceBands>
<targetDataType>int32</targetDataType>
<targetScalingStr>Truncate</targetScalingStr>
<targetNoDataValue>-32768</targetNoDataValue>
</parameters>
</node>
<node id="Write">
<operator>Write</operator>
<sources>
<sourceProduct refid="Convert-Datatype"/>
</sources>
<parameters class="com.bc.ceres.binding.dom.XppDomElement">
<file>/tmp/tmpo3kir6pz_input/S3A_OL_1_EFR____20210301T214636_20210301T214936_20210301T233636_0179_069_086_2700_LN1_O_NR_002_RADIANCE_1.tif</file>
<formatName>GeoTIFF-BigTIFF</formatName>
</parameters>
</node>
<applicationData id="Presentation">
<Description/>
<node id="Read">
<displayPosition x="20.0" y="134.0"/>
</node>
<node id="Reproject">
<displayPosition x="201.0" y="110.0"/>
</node>
<node id="Convert-Datatype">
<displayPosition x="330.0" y="112.0"/>
</node>
<node id="Write">
<displayPosition x="512.0" y="99.0"/>
</node>
</applicationData>
</graph>
Okay, I found the reason for this.
After the Convert-Datatype operator, the product has on more band compared to SNAP7. It is the quality_flags band.
This was introduced by resolving this issue: [SNAP-1232] Masks not correctly copied by ProductUtils.copyProductNodes method
The Convert-Datatype operator uses the ProductUtils.copyProductNodes method. That’s why it was affected by this. And this is actually good.
This band caused the BIGGeoTiff writer to switch to double. The flag band unsigned int32 and to handle this data properly we had to use the double data type. Exact reasons I can’t remember anymore. And right now, I’m questioning this.
This means for your graph that you need to add a subset operator after the conversion operation to include only the bands you like to have.
With the graph attached you should get the expected results.
updatedGraph.xml (2.7 KB)
Btw., why do you use Bicubic resampling and the change the datatype to integer? It seems you don’t need the accuracy. You could switch to Nearest or Bilinear. This could save you processing time.
Thank you, this makes sense. You must have made the data type change because geotiffs cannot handle multiple datatypes in a single file.
Why do you think the accuracy of Bicubic resampling is not needed?
Also yes, it’s pretty strange to put quality flags into a floating-point band. But if you needed uint32, then all output band would need that type, which would blow up the file size. Other dataset providers usually create multiple QA bands in such cases.
I saw some reference to a new format that will replace BEAM soon. How can I monitor its progress? We need to rewrite our entire S3 pipeline anyway.
You can install an early alpha version as plugin.
There is also a github repoitory for the development: bcdev/snap-zarr: Prototype implementation of a SNAP reader/writer using Zarr for storing the data (github.com)
This will be moved to snap-engine when done.
This snap-zarr implementation makes use of:
bcdev/jzarr: A Java implementation of the API offered by the Python zarr package (github.com)
BEAM-DIMAP will be still supported for a while. The capability to read the data will not be disabled. Only the writing of new data, will be disabled at some point in time.
1 Like
Yes, zarr is a better solution. Do you happen to have an estimate when the plugin will be out of alpha?
It is planned to have a version ready for the next release. This shall be feature complete. This will not yet replace BEAM-DIMAP. This might happen in the version afterwards.
The next release is scheduled for some time at Mid 2021.