I’m curious as to what sort of pre-processing methods/resampling are standard practice to make SAR imagery less massive to work with without losing too much data.
On my PC - simply exporting a Geotiff of three combined IW_GRDH sentinel tiles to cover part of the waters to the west (all corrections were applied separately - not included in this count) took about 24 hours. Filesize for a stripped down, single polarity geotiff is 12Gb, and we need an additional three tiles to complete the picture close in to our coastline - i’d say around 7-9 in total for all our waters which is about 30+Gb.
On the server we have setup for processing - running basic window and decibel threshold settings but with a medium cluster size for larger oil slick detection on a single sentinel tile took 15 days. Again - all the processing had already been applied.
It sounds like there could be an issue with the GeoTIFF-writer, @lveci do you remember if this is a known issue?
Could you show the processing graph and the input product?
Thanks for the replies - the geotiff was not created through a graph - this was a series of three sentinel GRD tiles that i had first corrected and then used the level slice assembly to combine into a single dim. It is approximately 12GB.
At this stage I tried to export as a Geotiff using the user interface.
File - Export - Geotiff. >>
or maybe i used Geotiff/Big Tiff - sorry I cannot remember. I know you can right click on the screen to and export view as image as well as Geotiff.
I tried to export a large image (two assembled GRD-slices in 10m pixel-size, ~10GB) into a GeoTIFF and the operation seemed to be extremely slow. I wonder what is going on, it’s not that writing a flat binary band should be time-consuming. This was via the export-menu, right-clicking on the image and saving as GeoTIFF produced an rather uninformative error-message. @marpet are you aware of such issues with the GeoTIFF-writer?
I just mad a little test. I’ve upsampled Landsat8 to the panchromatic resolution and stored it to BEAM-DIMAP. The resulting data amount is ~5GB. This took ~2:30 minutes. I think this is pretty reasonable.
Now I did the same but have chosen BIGGeoTIFF(with compression) as the storage format. This took ~5:20 minutes, without compression it took 3:40 minutes.
Here is the overview:
Format Time Size
dimap 2:30 5.1GB
bigtiff(compression) 5:20 3.2GB
bigtiff(no compression) 3:44 5.1GB
So yes, GeoTiff is slower but it is not totally out of scope.
I’ve updated the numbers. The first ones I’ve posted were not measured under equal conditions.
I tried this with a single zipped S1a image, File – export – bigtiff completed in approximately 2 mins.
Im trying it now with a slice assembly of three S1a images in .dim format with radiometric, speckle and terrain corrections already applied. The export is 1hr 10mins in and still at 0% in ‘writing band’ .
Edit: Still at 0% approx 4hrs 10 mins
Edit 2: moved to 1% after 4hrs 30mins
I experienced the same with 2 concatenated IW GRD slices - way too slow.