Expected execution times in TOPS interferometry?

YRauste · December 21, 2015, 3:32pm

Trying to compute coherence between pairs of Sentinel-1 IW images. I have followed the work flow described in the tutorial called “Sentinel-1 Toolbox - TOPS Interferometry Tutorial”, version May 2015 (by Luis Veci). It seems I can produce coherence images and re-project them to UTM. My challenge is that all the processing steps combined requires a total execution time that is close to a full working day. The processing steps require about the following execution times (from the messages at the end of the step in the SNAP GUI):

co-registration 3 minutes (times 3 sub-swaths)
interferogram 18-30 minutes(times 3 sub-swaths)
de-bursting 20-32 minutes (times 3 sub-swaths)
TOPS combination 62-64 minutes
Multi-looking 36(?) - 133 minutes
Terrain correction 1 minute
Are these execution times as expected on a fairly modern computer (Intel Core-i5-4570 CPU with 4 cores, 3.2 GHz, 16 GB of memory, running under Windows-7 Enterprise operating system) ? I even adopted the practice of relaunching SNAP after each step to prevent possible memory allocation problems connected with multiple open images in the GUI. If the processing time cannot be reduced, time-series with coherence data becomes a bit problematic. My trial with the graph builder stopped when I could not find the same co-registration dialogue as used on the GUI side.

Best regards
Yrjo Rauste

mengdahl · December 21, 2015, 3:49pm

Hello Yrjö,

The majority of the processing-time is spent on I/O-operations, therefore having a rapid SSD hard disk makes a huge difference in performance. You can reduce the number of I/O-operations by grouping some of your operations together in the Graph Builder - this way the step of writing and re-reading an intermediate product is skipped.

I do not think it’s necessary to close SNAP between operations, the memory-management is tiled and having a full tile-cache does not matter as the oldest tiles are just purged when new tiles are put in the cache.

Marcus

mfoumelis · December 21, 2015, 4:03pm

Indeed, as Marcus described, running them on SSD drives through the graph builder (without intermediate products) saves a lot of time.
You may also consider masking out water bodies during the TOPS coregisration step (if applicable).

lveci · December 21, 2015, 9:17pm

The coregistration dialog is made up of a graph with 4 operators in the case of automatic coregisteration. In the graph builder, you will need to assemble the coregistration with the same operators for example create stack, gcp selection and warp.

For S1 TOPS coregistration, these graphs already exist in the Graph Builders graph menu found at the top of the graph builder dialog.

YRauste · December 22, 2015, 10:28am

Hello Luis et al,

Thank you for your helpful and frank responses. I found the ready-made graph for co-registration: “TOPSAR Coreg Interferogram IW All Swaths.xml”. When trying to use it, the modules TOPSAR-Split(2)…TOPSAR-Split(6) seem not to have the right choices (or any choices for that matter) of sub-swath and polarization. The modules TOPSAR-Split and TOPSAR-Split(2) have the same choices as the corresponding dialogue on the GUI side. The consequence is (I think this is the consequence from missing parameters) that I cannot run the graph (red text: org.esa.snap.core.graph.GraphException). I got the same error message also on the GUI side after I had managed to input my external DEM (and switching off the error message that the area is outside the SRTM DEM area). I changed my test image pair to Finland (above 60 degrees North) and generated the DEM geotiff file, but this should have no effect on the graph, I guess. My DEM file is as the instructions require: geotiff, Palte-Carre projection, elevations in meters (signed 16-bit integers image data type, even though the instructions don’t say anything about image data type).

Ciao
Yrjo

lveci · December 22, 2015, 2:01pm

Did you assign proper inputs to the readers. There are two readers and by default both of them may automatically be set to the currently selected product.

YRauste · December 22, 2015, 2:25pm

Yes, I assigned both readers to their corresponding images. I noticed the somewhat illogical default behavior of assigning two readers to a single image.

Ciao
Yrjo

YRauste · December 23, 2015, 11:18am

Tried today the new version of Sentinel-1 toolbox, but no improvement. Then I decided to try again with the Mexican dataset and the big graph. It succeeded, and the execution time dropped to under 3 hours. This could be considered only as a partial success because the second image pair stopped with an error message: “Back-Geocoding: computeSlavePixPos: GC overhead limit exceeded”. The search function did not find this error message in the help, so I gave up this exercise.

There are two differences between my Finnish and Mexican test datasets:

in Mexico single-pol SSV and in Finland dual-pol SDV,
in Mexico the 3arccsec SRTM DEM and in Finland a quality DEM in geotiff format strictly following the documented requirements for the S1tbx external DEM.

Even though the dual-pol data makes the interferometry graph a real monster, my my guess is that the real problem is the refusal of the Sentinel-1 toolbox to ingest the external DEM - even though it conforms to all requirements the documentation manges to specify. In my opinion, the toolbox should check that an external DEM is valid (and can be read) when it accepts the file name in the input dialogue. I have the impression that this is not the case in the current version of the Sentinel-1 toolbox.

Just in case there might be some undocumented further requirements on an external DEM in geotiff format, I add here diagnostic listings (tiffdump of libtiff and listgeo of libgeotiff) of the properties of my DEM:

tiffdump:

Hdem20mLatLongC.tif:
Magic: 0x4949 Version: 0x2a
Directory 0: offset 8 (0x8) next 0 (0)
ImageWidth (256) SHORT (3) 1<19894>
ImageLength (257) SHORT (3) 1<9392>
BitsPerSample (258) SHORT (3) 1<16>
Compression (259) SHORT (3) 1<1>
Photometric (262) SHORT (3) 1<1>
StripOffsets (273) LONG (4) 9392<75502 115290 155078 194866 234654 274442 314230
354018 393806 433594 473382 513170 552958 592746 632534 672322 712110 751898 79
1686 831474 871262 911050 950838 990626 …>
SamplesPerPixel (277) SHORT (3) 1<1>
RowsPerStrip (278) SHORT (3) 1<1>
StripByteCounts (279) LONG (4) 9392<39788 39788 39788 39788 39788 39788 39788 39
788 39788 39788 39788 39788 39788 39788 39788 39788 39788 39788 39788 39788 3978
8 39788 39788 39788 …>
PlanarConfig (284) SHORT (3) 1<1>
SampleFormat (339) SHORT (3) 1<2>
33550 (0x830e) DOUBLE (12) 3<0.000180001 0.000180001 0>
33922 (0x8482) DOUBLE (12) 6<0 0 0 22.5734 62.6333 0>
34735 (0x87af) SHORT (3) 32<1 1 0 7 1024 0 1 2 1025 0 1 1 2048 0 1 4326 2049 347
37 7 0 2054 0 1 9102 …>
34736 (0x87b0) DOUBLE (12) 2<298.257 6.37814e+006>
34737 (0x87b1) ASCII (2) 8<WGS 84|\0>

listgeo:

Geotiff_Information:
Version: 1
Key_Revision: 1.0
Tagged_Information:
ModelTiepointTag (2,3):
0 0 0
22.573407 62.6332688 0
ModelPixelScaleTag (1,3):
0.000180001 0.000180001 0
End_Of_Tags.
Keyed_Information:
GTModelTypeGeoKey (Short,1): ModelTypeGeographic
GTRasterTypeGeoKey (Short,1): RasterPixelIsArea
GeographicTypeGeoKey (Short,1): GCS_WGS_84
GeogCitationGeoKey (Ascii,7): “WGS 84”
GeogAngularUnitsGeoKey (Short,1): Angular_Degree
GeogSemiMajorAxisGeoKey (Double,1): 6378137
GeogInvFlatteningGeoKey (Double,1): 298.257224
End_Of_Keys.
End_Of_Geotiff.

GCS: 4326/WGS 84
Datum: 6326/World Geodetic System 1984
Ellipsoid: 7030/WGS 84 (6378137.00,6356752.31)
Prime Meridian: 8901/Greenwich (0.000000/ 0d 0’ 0.00"E)

Corner Coordinates:
Upper Left ( 22d34’24.27"E, 62d37’59.77"N)
Lower Left ( 22d34’24.27"E, 60d56’33.72"N)
Upper Right ( 26d 9’15.65"E, 62d37’59.77"N)
Lower Right ( 26d 9’15.65"E, 60d56’33.72"N)
Center ( 24d21’49.96"E, 61d47’16.74"N)

Ciao
Yrjo