View TIFF images (LZW compression) with ESA SNAP

FrancescoAdesso · August 9, 2018, 8:53am

I did a conversion with Irfanview using the Windows Command Line (advancedbatch) from TIFF Packbit to TIFF LZW. When I open the file in ESA SNAP it loads extremely slow. I have the same Picture which was converted in the same way with Irfanview last year which works just fine. In the Metadata the only difference is the RowsPerStrip. Do you have an idea why it loads extremely slow?

RowsPerStrip : 1024 (TIFF not loading properly)

RowsPerStrip: 1 (TIFF loading ok)

marpet · August 10, 2018, 1:23pm

Could you show what is written in the metadata within SNAP Desktop.
Expand all nodes and take a screenshot. Like this one:

FrancescoAdesso · August 10, 2018, 1:50pm

workswell

This one is the metadata of the TIFF causes Trouble:
doesntworkwell

marpet · August 10, 2018, 2:39pm

One more question. Sorry, should have asked this before.

What shows SNAP when you open the information window (Analysis / Information). Especially I’m interested in the preferred tile width and height. Is this is not in line with the striping of your data, this could be a reason for the slowness.

FrancescoAdesso · August 13, 2018, 3:52pm

Hallo,
ich habe gesehen, wir können auch auf Deutsch wechseln…! Hier sind nochmal Screenshots von dem Info Fenster. Beide Dateien unterscheiden sich hier in keinem Punkt. Können Sie das ‘Is this is not in line with the striping of your data, this could be a reason for the slowness’ näher beschreiben?

FrancescoAdesso · August 13, 2018, 3:59pm

marpet · August 14, 2018, 6:01am

Hi Francesco,
ja wir könnten auf deutsch weiter reden, aber dann würden uns viele anedere nicht verstehen.
So, it is better to stick to English, then everyone can benefit from the discussion.

Can you send me the two files? Maybe as links in a PM? If not, I can provide a location where you can upload them.

The differences in the striping of the Tiff file and the tile size in SNAP can be a reason for the slowness because of the way the data is read. If a 320x320 tile shall be read by SNAP all 1024 rows must be unpacked and for the next tile again, and again… There is no caching implemented.
But actually these values should be inline and then only the data for one tile is unpacked. So this might be a bug in the reader. Your data will be helpful to reproduce the behaviour and fix the issue.

FrancescoAdesso · August 14, 2018, 9:22am

Dear Marco,

could you please provide a upload Location then?

marpet · August 15, 2018, 8:11am

I had a look at your files and it behaves as you say. But I couldn’t figure out why the second one is slow.
We check if the tiling is good if it is not we compute our own tiling size and use it. This works quite well if the data is not compressed, but it is probably contradictive when compression is used. We consider a tiling as bad if the tiles span the full width or height of the image or just one line or row. In the terms of memory management and read performance, this is bad. For images this usually doesn’t matter, that’s probably why IrfanView does this, but for EO data this can be a difference.
However, even if I use the predefined tiling by the files it is only slightly better.
I tried to change how the data is read and this gave much better results. The slow image was displayed after 2 seconds instead of 20-30 seconds. Unfortunately, the new implementation results in other errors. So, this needs more testing and validation but at the moment I can’t spend more time on this. I’ve created a ticket in our issue tracker so that we can follow the problem later.

Instead of IrfanView you could use GDAL for the conversion.
I think this page can help you with it: https://geoserver.geo-solutions.it/edu/en/raster_data/processing.html
There you can specify a tiling. Maybe something like 500x500.
This will also preserve the geo-information of the GeoTiff file. If your initial images have it.

kygoh · June 8, 2019, 6:10am

I found missing tiles on the Navigation window and Image window when I used the SNAP Desktop NB Application to view a BigTiff file with the following properties:

ImageWidth 7044
ImageHeight 15933
Compression LZW
TileWidth 256
TileHeight 256
BitsPerSample 16, 16, 16, 16, 16

From the messages.log a few NullPointerExceptions were thrown.

Tracing the code to it.geosolutions.imageioimpl.plugins.tiff.TIFFLZWDecompressor, they were caused by the following 3 situations:

NullPointerException at line 278 below:

github.com

senbox-org/imageio-ext/blob/master/plugin/tiff/src/main/java/it/geosolutions/imageioimpl/plugins/tiff/TIFFLZWDecompressor.java#L278


            for (int i=0; i < maxIndex; i++) {
                dstData[dstIndex++] = string[i];
            }
        }
    }
    
    /**
     * Add a new string to the string table.
     */
    public void addStringToTable(byte oldString[], byte newString) {
	int length = oldString.length;
	byte string[] = new byte[length + 1];
	System.arraycopy(oldString, 0, string, 0, length);
	string[length] = newString;
	
	// Add this new String to the table
	stringTable[tableIndex++] = string;
	
	if (tableIndex == 511) {
	    bitsToGet = 10;
	} else if (tableIndex == 1023) {

because stringTable[oldCode] passed from line 215 of the caller method is null.

github.com

senbox-org/imageio-ext/blob/master/plugin/tiff/src/main/java/it/geosolutions/imageioimpl/plugins/tiff/TIFFLZWDecompressor.java#L215


		    break;
		}


		writeString(stringTable[code]);
		oldCode = code;
	    } else {
		if (code < tableIndex) {
		    string = stringTable[code];


		    writeString(string);
		    addStringToTable(stringTable[oldCode], string[0]); 
		    oldCode = code;
		} else {
		    string = stringTable[oldCode];
		    string = composeString(string, string[0]);
		    writeString(string);
		    addStringToTable(string);
		    oldCode = code;
		}
	    }
	}

NullPointerException at line 219 below because stringTable[oldCode] from line 218 is null:

github.com

senbox-org/imageio-ext/blob/master/plugin/tiff/src/main/java/it/geosolutions/imageioimpl/plugins/tiff/TIFFLZWDecompressor.java#L218-L219


		    string = stringTable[oldCode];
		    string = composeString(string, string[0]);

NullPointerException at line 265 below:

github.com

senbox-org/imageio-ext/blob/master/plugin/tiff/src/main/java/it/geosolutions/imageioimpl/plugins/tiff/TIFFLZWDecompressor.java#L265


	
	tableIndex = 258;
	bitsToGet = 9;
    }


    /**
     * Write out the string just uncompressed.
     */
    public void writeString(byte string[]) {
        if(dstIndex < dstData.length) {
            int maxIndex = Math.min(string.length,
				    dstData.length - dstIndex);


            for (int i=0; i < maxIndex; i++) {
                dstData[dstIndex++] = string[i];
            }
        }
    }
    
    /**
     * Add a new string to the string table.

because stringTable[code] passed from line 214 of the caller method is null:

github.com

senbox-org/imageio-ext/blob/master/plugin/tiff/src/main/java/it/geosolutions/imageioimpl/plugins/tiff/TIFFLZWDecompressor.java#L214


		if (code == 257) {
		    break;
		}


		writeString(stringTable[code]);
		oldCode = code;
	    } else {
		if (code < tableIndex) {
		    string = stringTable[code];


		    writeString(string);
		    addStringToTable(stringTable[oldCode], string[0]); 
		    oldCode = code;
		} else {
		    string = stringTable[oldCode];
		    string = composeString(string, string[0]);
		    writeString(string);
		    addStringToTable(string);
		    oldCode = code;
		}
	    }

It appears to me that TIFFLZWDecompressor.decode() may possibly be run concurrently by multiple TIFFImageReader.decodeTile(int ti, int tj, int band) calls causing stringTable in TIFFLZWDecompressor class to have null values.

Is it related SNAP-941?