Trade off in processing time v resolution

Microwave Toolbox
#1

Hi All,

I’m curious as to what sort of pre-processing methods/resampling are standard practice to make SAR imagery less massive to work with without losing too much data.

On my PC - simply exporting a Geotiff of three combined IW_GRDH sentinel tiles to cover part of the waters to the west (all corrections were applied separately - not included in this count) took about 24 hours. Filesize for a stripped down, single polarity geotiff is 12Gb, and we need an additional three tiles to complete the picture close in to our coastline - i’d say around 7-9 in total for all our waters which is about 30+Gb.

On the server we have setup for processing - running basic window and decibel threshold settings but with a medium cluster size for larger oil slick detection on a single sentinel tile took 15 days. Again - all the processing had already been applied.

Advice appreciated,

Thanks,

Conor.

1 Like
#2

It sounds like there could be an issue with the GeoTIFF-writer, @lveci do you remember if this is a known issue?

#3

Could you show the processing graph and the input product?

#4

Thanks for the replies - the geotiff was not created through a graph - this was a series of three sentinel GRD tiles that i had first corrected and then used the level slice assembly to combine into a single dim. It is approximately 12GB.

At this stage I tried to export as a Geotiff using the user interface.

File - Export - Geotiff. >>

or maybe i used Geotiff/Big Tiff - sorry I cannot remember. I know you can right click on the screen to and export view as image as well as Geotiff.

Exporting Sentinel 2a Image to GeoTiff/BigTiff
#5

I tried to export a large image (two assembled GRD-slices in 10m pixel-size, ~10GB) into a GeoTIFF and the operation seemed to be extremely slow. I wonder what is going on, it’s not that writing a flat binary band should be time-consuming. This was via the export-menu, right-clicking on the image and saving as GeoTIFF produced an rather uninformative error-message. @marpet are you aware of such issues with the GeoTIFF-writer?

#6

I just mad a little test. I’ve upsampled Landsat8 to the panchromatic resolution and stored it to BEAM-DIMAP. The resulting data amount is ~5GB. This took ~2:30 minutes. I think this is pretty reasonable.
Now I did the same but have chosen BIGGeoTIFF(with compression) as the storage format. This took ~5:20 minutes, without compression it took 3:40 minutes.
Here is the overview:

Format                    Time     Size
dimap                     2:30     5.1GB
bigtiff(compression)      5:20     3.2GB
bigtiff(no compression)   3:44     5.1GB

So yes, GeoTiff is slower but it is not totally out of scope.

1 Like
#7

I’ve updated the numbers. The first ones I’ve posted were not measured under equal conditions.

#8

I tried this with a single zipped S1a image, File – export – bigtiff completed in approximately 2 mins.

Im trying it now with a slice assembly of three S1a images in .dim format with radiometric, speckle and terrain corrections already applied. The export is 1hr 10mins in and still at 0% in ‘writing band’ .

Edit: Still at 0% approx 4hrs 10 mins
Edit 2: moved to 1% after 4hrs 30mins

1 Like
#9

I experienced the same with 2 concatenated IW GRD slices - way too slow.

1 Like