Coherence has gaps near SRTM 3 tile bounds

j08lue · May 31, 2019, 10:03am

I compute coherence over a small country, using gpt.

However, at the edges of SRTM 3Sec tiles, I get weird gaps in the output data. Here is the graph:

<graph id="Graph">
  <version>1.0</version>
  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>/mnt/downloaded/S1B_IW_SLC__1SDV_20190506T052401_20190506T052431_016119_01E53A_D724.zip</file>
    </parameters>
  </node>
  <node id="ProductSet-Reader">
    <operator>ProductSet-Reader</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <fileList>/mnt/downloaded/S1A_IW_SLC__1SDV_20190430T052344_20190430T052411_027015_030ABC_DCDB.zip,/mnt/downloaded/S1A_IW_SLC__1SDV_20190430T052408_20190430T052436_027015_030ABC_3709.zip,/mnt/downloaded/S1A_IW_SLC__1SDV_20190430T052434_20190430T052501_027015_030ABC_0D07.zip,/mnt/downloaded/S1A_IW_SLC__1SDV_20190430T052459_20190430T052526_027015_030ABC_85BE.zip</fileList>
    </parameters>
  </node>
  <node id="SliceAssembly">
    <operator>SliceAssembly</operator>
    <sources>
      <sourceProduct.3 refid="ProductSet-Reader"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <selectedPolarisations>VV</selectedPolarisations>
    </parameters>
  </node>
  <node id="TOPSAR-Split">
    <operator>TOPSAR-Split</operator>
    <sources>
      <sourceProduct refid="SliceAssembly"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <subswath>IW3</subswath>
      <selectedPolarisations/>
      <firstBurstIndex>1</firstBurstIndex>
      <lastBurstIndex>999999</lastBurstIndex>
      <wktAoi/>
    </parameters>
  </node>
  <node id="TOPSAR-Split(2)">
    <operator>TOPSAR-Split</operator>
    <sources>
      <sourceProduct refid="Read"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <subswath>IW3</subswath>
      <selectedPolarisations/>
      <firstBurstIndex>1</firstBurstIndex>
      <lastBurstIndex>999999</lastBurstIndex>
      <wktAoi/>
    </parameters>
  </node>
  <node id="Apply-Orbit-File">
    <operator>Apply-Orbit-File</operator>
    <sources>
      <sourceProduct refid="TOPSAR-Split"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <orbitType>Sentinel Precise (Auto Download)</orbitType>
      <polyDegree>3</polyDegree>
      <continueOnFail>false</continueOnFail>
    </parameters>
  </node>
  <node id="Apply-Orbit-File(2)">
    <operator>Apply-Orbit-File</operator>
    <sources>
      <sourceProduct refid="TOPSAR-Split(2)"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <orbitType>Sentinel Precise (Auto Download)</orbitType>
      <polyDegree>3</polyDegree>
      <continueOnFail>false</continueOnFail>
    </parameters>
  </node>
  <node id="Back-Geocoding">
    <operator>Back-Geocoding</operator>
    <sources>
      <sourceProduct refid="Apply-Orbit-File"/>
      <sourceProduct.1 refid="Apply-Orbit-File(2)"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <demName>SRTM 3Sec</demName>
      <demResamplingMethod>BILINEAR_INTERPOLATION</demResamplingMethod>
      <externalDEMFile/>
      <externalDEMNoDataValue>0.0</externalDEMNoDataValue>
      <resamplingType>BILINEAR_INTERPOLATION</resamplingType>
      <maskOutAreaWithoutElevation>true</maskOutAreaWithoutElevation>
      <outputRangeAzimuthOffset>false</outputRangeAzimuthOffset>
      <outputDerampDemodPhase>true</outputDerampDemodPhase>
      <disableReramp>false</disableReramp>
    </parameters>
  </node>
  <node id="Coherence">
    <operator>Coherence</operator>
    <sources>
      <sourceProduct refid="Back-Geocoding"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <cohWinAz>5</cohWinAz>
      <cohWinRg>20</cohWinRg>
      <subtractFlatEarthPhase>true</subtractFlatEarthPhase>
      <srpPolynomialDegree>5</srpPolynomialDegree>
      <srpNumberPoints>1001</srpNumberPoints>
      <orbitDegree>3</orbitDegree>
      <squarePixel>true</squarePixel>
      <subtractTopographicPhase>false</subtractTopographicPhase>
      <demName>SRTM 3Sec</demName>
      <externalDEMFile/>
      <externalDEMNoDataValue>0.0</externalDEMNoDataValue>
      <externalDEMApplyEGM>true</externalDEMApplyEGM>
      <tileExtensionPercent>100</tileExtensionPercent>
    </parameters>
  </node>
  <node id="TOPSAR-Deburst">
    <operator>TOPSAR-Deburst</operator>
    <sources>
      <sourceProduct refid="Coherence"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <selectedPolarisations/>
    </parameters>
  </node>
  <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="TOPSAR-Deburst"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>/mnt/data/S1_20190430_20190506_D_168.IW3.dim</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
  </node>
  <applicationData id="Presentation">
    <Description/>
    <node id="SliceAssembly">
      <displayPosition x="165.0" y="61.0"/>
    </node>
    <node id="TOPSAR-Split">
      <displayPosition x="300.0" y="61.0"/>
    </node>
    <node id="TOPSAR-Split(2)">
      <displayPosition x="293.0" y="138.0"/>
    </node>
    <node id="Apply-Orbit-File">
      <displayPosition x="414.0" y="62.0"/>
    </node>
    <node id="Apply-Orbit-File(2)">
      <displayPosition x="413.0" y="137.0"/>
    </node>
    <node id="Back-Geocoding">
      <displayPosition x="561.0" y="138.0"/>
    </node>
    <node id="Coherence">
      <displayPosition x="694.0" y="137.0"/>
    </node>
    <node id="TOPSAR-Deburst">
      <displayPosition x="816.0" y="138.0"/>
    </node>
    <node id="Write">
      <displayPosition x="953.0" y="138.0"/>
    </node>
    <node id="Read">
      <displayPosition x="15.0" y="136.0"/>
    </node>
    <node id="ProductSet-Reader">
      <displayPosition x="15.0" y="61.0"/>
    </node>
  </applicationData>
</graph>

S1_20190430_20190506_D_168.IW3.gpt.xml (5.8 KB)

And here is the (terrain-corrected) output, with the northmost SRTM tile (39_01) in red and the southern (39_02) in purple. Please note that the gaps occur already in the coherence output, not only after terrain correction.

I get this on many products and always at the edge of neighboring SRTM 3 tiles.

The freakiest part of this is that the missing data gaps are different for subsequent runs of the same graph, like this:

This is on Linux (https://github.com/DHI-GRAS/docker-esa-snap) with the newest SNAP and plugin updates and with 256 GB of RAM and 48 cores.

falahfakhri · May 31, 2019, 10:13am

You have used SRTM 3 sec, and this DEM has problem of connection, and might not be downloaded at all,

Try to use SRTM 1sec, almost 30 m,

j08lue · May 31, 2019, 10:14am

FWIW, here is the terminal output from gpt:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder '.'.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.engine_utilities.download.downloadablecontent.DownloadableContentImpl: http retrieving http://aux.sentinel1.eo.esa.int/POEORB/2019/05/20/S1A_OPER_AUX_POEORB_OPOD_20190520T120641_V20190429T225942_20190501T005942.EOF
INFO: org.esa.snap.engine_utilities.download.downloadablecontent.DownloadableContentImpl: http retrieving http://aux.sentinel1.eo.esa.int/POEORB/2019/05/26/S1A_OPER_AUX_POEORB_OPOD_20190526T120747_V20190505T225942_20190507T005942.EOF
INFO: org.esa.snap.engine_utilities.download.downloadablecontent.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/S1B/2019/05/S1B_OPER_AUX_POEORB_OPOD_20190526T110527_V20190505T225942_20190507T005942.EOF.zip
WARNING: org.jlinda.core.Baseline: Max. error bperp modeling at 3D datapoints: 3.3548212415907557m
======
Master: 30Apr2019
Slave: 30Apr2019 perp baseline: 0.0 temp baseline: 0.0
Slave: 06May2019 perp baseline: 27.669243 temp baseline: -6.000221

======
Master: 06May2019
Slave: 30Apr2019 perp baseline: -26.55908 temp baseline: 6.000221
Slave: 06May2019 perp baseline: 0.0 temp baseline: 0.0

INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/srtm_39_01.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://srtm.csi.cgiar.org/srt-zip/srtm_v41/srtm_data_geotiff/srtm_39_01.zip on http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/srtm_39_01.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_39_01.zip
....10%....20%....30%....40%....50%.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/srtm_39_02.zip
.WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:http://srtm.csi.cgiar.org/srt-zip/srtm_v41/srtm_data_geotiff/srtm_39_02.zip on http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/srtm_39_02.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_39_02.zip
..60%....70%....80%....90% done.

j08lue · May 31, 2019, 10:18am

Thanks @falahfakhri, I’ll try that.

I also tried fixing the SRTM download URL in the snap.auxdata.properties (as in SRTM not downloading) like sed -i 's|http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/|http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/|g' /usr/local/snap/etc/snap.auxdata.properties and the WARNING went away.

Furthermore, I tried pre-downloading the SRTM tiles in question and placing them in ~/.snap/auxdata/dem/SRTM 3Sec (still zipped), but they still get downloaded every time.

Why is the DEM even used? The setting seems to be only related to subtractTopographicPhase, which is not activated.

j08lue · May 31, 2019, 10:30am

NB Since the S1 products are clipped along-orbit at varying location, I sometimes need to assemble several along-orbit slices for the first date while I need only one for the second date - like in the above graph.

j08lue · May 31, 2019, 11:06am

With SRTM 1Sec HGT, the gaps are gone:

falahfakhri · May 31, 2019, 11:09am

So the SRTM 1 sec is the solution as I mentioned in my previous post Source of the psot

j08lue · May 31, 2019, 11:17am

Right. We still don’t know what’s wrong with (the algorithm’s use of) SRTM 3Sec, though.

j08lue · May 31, 2019, 11:19am

Btw, there might also be an issue with my selection of inputs for the two dates: Either the first one should include more, or the second one fewer slices. Like here (same thing, different date), where 1 is the single slice I use for the first date and 2-5 are the images in the SliceAssembly:

But that is a different issue.

j08lue · June 3, 2019, 9:14am

Just for the record - we have a workaround, but no solution for this issue yet.