BandSelect produces multisize product out of same-size bands


#1

When I try to save a band set (B2,B3,B4,B8) of Sentinel-2 image as GeoTiff-BigTIFF, I get an error message “Product writer is unable to write this product as ‘GeoTIFF-BigTIFF’: Cannot write multisize products. Consider resampling the product first.” When I save as BEAM-DIMAP, I get a set of 4 ENVI images that are perfectly the same size (and the same geographic location). Very similar errors are produced also in graph builder when trying to limit the scope to bands of a certain resolution.

I am using SNAP 5.0.8 and S2TBX 5.0.7.

Ciao
Yrjö


#2

Hi Yrjö, I think that TIFF does not support bands with differing pixel sizes - therefore you need to resample the bands into a common resolution first if you want to stick them into a single TIFF.


#3

Bands B2,B3,B4,B8 are all 10-m bands - no differing pixel sizes. This is really a bug in the way SNAP handles Sentinel-2 images. If some obscure ghost components (e.g. cloud masks or detector masks) are of different sizes, they should not affect the output of BandSelect operator.

I know that Resample operator can be used as a way around this bug. Unfortunately this alternative puts a heavy penalty on elapsed time, CPU time, and memory footprint. The situation is worsened in graphs, where the Resample operator seems to be needed practically after every operation you apply to the selected, same-size bands. Because the Resample operator is just a dummy nuisance step imposed by the SNAP architecture/implementation (the resulting image data is bit-to-bit identical to that what I requested - and what I got in the BEAM-DIMAP alternative), the need for this type of resource-consuming null-operators should be eliminated. For instance in the F-TEP case, we have a tile of Sentinel-2 data at 10-m resolution. Adding two Resample operations in a re-processing graph - just as a way around some SNAP artefacts - puts the end user waiting tens of minutes more.

Ciao
Yrjö


#4

Right Yrjö, this sounds like a bug. @marpet @Fabrizio_Ramoino is this a known issue?

If this issue is still present in the 6.0 beta it should definitely be solved for the 6.0 official release.


#5

Hi Yrjö

actually you need the resampling only once at the beginning of your graph and the output can be used in the successive steps.
And if you only deal with in your chain with the 10m bands it adds no, or at least only a little extra processing time, because, as you already said, it is a no-op.

I tried the use case you have described in different ways and in 5.0.x and 6 BETA versions.
In the Graph Builder, on the command line and doing the steps separately in the GUI. And it works.
It is not slower then simply removing all bands and masks from the product. This can be easiest done, by unchecking from the context menu of the product the option “Group Nodes by Type”. Afterwards you can select all the nodes you don’t want and hit DEL-Key. Then you can export the product with out resampling. It takes the same time as when doing Resampling and BandSelect in the Graph Builder.

What slows done the export is the format. Big GeoTiff is slower.
For my scene it took:
DIMAP: 76 seconds
Big GeoTiff: 4 minutes
GeoTiff: 80 seconds

If you only want to have these 4 bands you have mentioned, you can use the ‘normal’ GeoTiff format, because it doesn’t exceed the 2GB barrier.

I’ve attached my graph which I have used on the command line with gpt.
S2_Resample_BandSelect.xml (1.5 KB)


#6

Hi Marco,

Thank you for your proposal to get around this bug. Your graph seems to do what it is supposed to do. My original aim (in a graph made by my colleagues) was to select bands and make a spatial subset over a bounding box of a polygon (using the Subset operator). Can you make a graph demonstrating this without including the Resample operator twice ?

Ciao
Yrjö


#7

Yes, I have it at hand. S2_Resample_Subset.xml (1.9 KB)

I only disabled the geoRegion property here because it didn’t match my current source product.


#8

Hello Marco,

I tried your second script with my dataset. The BEAM-DIMAP version went smoothly. When I switched to GeoTIFF-BigTIFF format I got again the error message:

“Error: [NodeId: Write] Product writer is unable to write this product as ‘GeoTIFF-BigTIFF’: Cannot write multisize products. Consider resampling the product first.”

Ciao
Yrjo


#9

You’re right. Probably I mixed something up while testing on SNAP 5.x.
But on SNAP 6 BETA it is working. Is it an option for you to try the current preview?


#10

I tested it with the gpt of the “current preview” (esa-snap_sentinel_unix_6_0-PREVIEW4.sh). Unfortunately the error message is the same as before.

Ciao
Yrjö


#11

I tried again. This time on Unix.

I’ve used the following graph, which is almost the same as before.
S2_Resample_Subset.xml (1.8 KB)

Is it possible that you have gpt on the path? You can type gpt --diag to see which version is invoked by a call to gpt.


Snap do not perform resample
#12

I hope this issue will be solved. But in general the recommendation is to do all the processing in BEAM-DIMAP and export in a separate step.


#13

I come back to testing later, but naturally I have gpt on PATH. The SNAP installation script creates a link to /usr/local/bin.

    tteyar@espvm30301:~/red/prog/fire/dun$ gpt --diag
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder '.'.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SNAP Release version 6.0-PREVIEW4
SNAP home: /opt/snap/bin//..
SNAP debug: null
SNAP log level: null
Java home: /opt/snap/jre
Java version: 1.8.0_102
Processors: 4
Max memory: 8.9 GB
Cache size: 1024.0 MB
Tile parallelism: 4
Tile size: 512 x 512 pixels
    
To configure your gpt memory usage:
Edit snap/bin/gpt.vmoptions

To configure your gpt cache size and parallelism:
Edit .snap/etc/snap.properties or gpt -c ${cachesize-in-GB}G -q ${parallelism} 
tteyar@espvm30301:~/red/prog/fire/dun$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
tteyar@espvm30301:~/red/prog/fire/dun$ 

The SNAP/gpt version seems to be the right one, and it seems to stay within its sand box using its own version of the java virtual machine (1.8.0_102), not the system-wide default (1.8.0.144).

Ciao
Yrjö


#14

Hello Marco,

Finally got to testing your latest (4.9.2017) graph. Unlike your screendump above, my result was the same error message as before ( …Cannot write multisize products.). Both on Windows (SNAP 5.0.8) and on Linux (SNAP 6.0.0).

Ciao
Yrjö

PS. In one of your postings above, you identify Big Geotiff as a source of slowing down processing. To clarify this, BIGTIFF as such is not a proble, the way SNAP writes BIGTIFF is. I compared gadlwarp with and without the command argument ‘-co “BIGTIFF=YES”’, and the excecution times were practically identical for a 9000x9000 image (actually BIGTIFF faster: 20.563 s, ClassicFIFF: 21.692 s).

PS. Marcus, if the recommendation is always to use BEAM-DIMAP and to add another gpt run for conversion, is this recommendation reflected in all tutorials and documentation of the graph engine of SNAP ?


#15

With saying that the Bigtiff is the cause for the slowness I just wanted to say that it is not caused by the resampling. And the slowness is caused by the the implementation within SNAP for sure.

The recommendation to use BEAM-DIMAP applies only for intermediate results and not for the final output. The final result can be written to any format you want. But for intermediate results BEAM-DIMAP is recommended in order to retain all necessary information.


#16

We should probably update help and tutorials so that this is clearly stated “best practice”. It should also be in a FAQ on the forum actually.