Inconsistent processing: shifting in some images

polar · July 31, 2018, 3:43pm

I am processing a large quantity of S1 data and have observed inconsistent processing in some images. For example, see the shift or skewing in the leftmost portion of the following animation:

This is a single S1 granule that I processed twice using the same version of SNAP, the same computer, the same processing graph, the same everything. The first image shows that the leftmost portion is shifted or skewed relative to reality (and most other images). Upon reprocessing the S1 zip file, the shift appears to be corrected.

I have observed this with approximately 25% of images I’ve processed for this particular scene. Reprocessing the images today seems to correct the skew, even though nothing has changed in terms of my software, etc.

This shift is a major problem in my research workflow, and manually examining and reprocessing images is not a viable option given the quantity of data I am processing.

Any idea what is going on?

lveci · July 31, 2018, 6:53pm

Could you list the product and the processing steps. How can the problem be reproduced?

ABraun · July 31, 2018, 7:20pm

how large is the shift (in pixels)?
Could it be that it is just a visual problem while the data is resampled by the GUI when zooming out? How does it look when you zoom in?

polar · July 31, 2018, 7:28pm

The product is S1A_EW_GRDM_1SSH_20151210T003102_20151210T003202_008972_00CDCA_7CED (also occurred with other 2015 and 2014 S1A data).

The processing steps are:

apply orbit file
calibration
terrain flattening
terrain correction
linearToFromdB

The TF and TC use an external DEM. The output is saved as a GeoTiff.

How it can be reproduced – I have no idea. I processed the image a second time today and the shift disappeared. The first time processing was yesterday.

To answer @ABraun’s question, it’s real. The offset is 500 m in the upper left corner.

Could it possibly be that the orbit file wasn’t correctly downloaded or applied?

ABraun · July 31, 2018, 7:29pm

alright, sorry for asking. But I had something similar once which turned out as just a visual shift at certain zoom levels.

polar · July 31, 2018, 7:32pm

Definitely a good question! I first suspected (and hoped) it may be visual, but unfortunately it’s real!

css · August 1, 2018, 9:59am

If you keep a copy of the shifted product, you may be able to verify that by checking the orbit information and comparing it with the same information in the correctly geolocated product. Look for Orbit_State_Vectors in SNAP’s abstracted metadata. Compare number of state vectors, times, positions. A bit tedious, but maybe something quickly stands out.
However, I suspect that if something goes wrong during the updating of the orbits, the effect on the TCed image would be more significant than what you are observing. Or it wouldn’t be observable at all if SNAP reverts back to the original orbits.

What DEM are you using? Are you sure it did not change between runs? Maybe SNAP cannot read it consistently?
Can you see whether the transition between shifted and unshifted portions of the image is sharp or smooth? Does it follow a straight line?

polar · August 1, 2018, 10:20pm

I deleted the original (shifted) image that I used to make the above gif, unfortunately. However, I’ve noticed this same error in another nearby product. Again, the shift disappears after reprocessing the image. It is more subtle here, and limited mainly to the rightmost side.

Here it is zoomed in:

You can see that the outermost edge is fixed.

The DEM I am using is the BEDMAP2 surface DEM, available here: http://data.pgc.umn.edu/gis/packages/quantarctica/Quantarctica3/TerrainModels/BEDMAP2/bedmap2_surface.tif

It’s coarse (1km) and probably doesn’t do a very good job, but there are few DEMS to choose from for Antarctica, and my main interest is in the ice shelves that are more or less flat. The DEM file is located on my computer (iMac Pro), so I don’t think there should be an issue with accessing it.

I checked the state vectors as suggested, and they look identical in the “shifted” and “non-shifted” images. Same number, and the few that I checked in detail had the same times, x_pos, etc.

Something else I noticed is that this region was not shifted in the GIF I showed in my first post, but is shifted in the second product. This leads me to believe it’s not a problem with the DEM itself, but perhaps how SNAP is ingesting it.

css · August 2, 2018, 4:51pm

If I am interpreting these images correctly, the shift is in the range direction. This would be consistent with an error in the height information used in the TC step. So DEM-related. But I cannot figure out any way to try to give cues as to what the source of the problem is.

throwaway-step-user · May 20, 2021, 5:57pm

Hi there, I know this is a +3 year old post, but I’m encountering this issue too.

I am on SNAP v8.0.3 running on ubuntu 16.04.

I am using this docker image for running snap but updated to v8 https://github.com/jncc/s1-ard-processor/blob/71fe73721064bd8df18aa618e310ae9d5d25b4f2/base/dockerfile

I am observing a pixel shift in the intermediate “TC” GeoTIFF file. See animated GIF of an area of interest around a major motorway feature where the spatial shift is obvious when comparing to an the OpenStreetMap base map.

Some further comments which may provide context:

I’ve processed other S1 scenes and there’s a spatial shift in those images too.
I have observed the shift in both ASCENDING and DESCENDING scenes.
The shift always occurs in the northern boundary of the scene and seems to only affect the northern most 2 km of the scene, south of that 2km band, there’s no “shift” observed.
The shift is not uniformily consistent across the entire northern edge. Some water features do not appear to be shifted.
- I’m using a custom Terrain file at (approx 10m cell size). Running gdalinfo on my DEM yields nothing obviously wrong though, see output:

Driver: GTiff/GeoTIFF
DTM_UK_10m_WGS84_CompImg_S1vers3.tif
Size is 133346, 101198
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,0905T175
            AUTHORITY["EPSG","7030"]],"working
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],0905T175
    AUTHORITY["EPSG","4326"]]"working
Origin = (-12.078822845297925,61.580952237851541)
Pixel Size = (0.000117820549665,-0.000117820549665)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND75807_03
Corner Coordinates:: "/stat
Upper Left  ( -12.0788228,  61.5809522) ( 12d 4'43.76"W, 61d34'51.43"N)1_metada
Lower Left  ( -12.0788228,  49.6577483) ( 12d 4'43.76"W, 49d39'27.89"N)
Upper Right (   3.6320762,  61.5809522) (  3d37'55.47"E, 61d34'51.43"N)75807_03
Lower Right (   3.6320762,  49.6577483) (  3d37'55.47"E, 49d39'27.89"N): "/stat
Center      (  -4.2233733,  55.6193502) (  4d13'24.14"W, 55d37' 9.66"N)1_metada
Band 1 Block=133346x1 Type=Float32, ColorInterp=Gray
  Description = Band 1
  NoData Value=-32768
  Metadata:
    ATTRIBUTETABLE_CHUNKSIZE=1000
    LAYER_TYPE=athematic

Finally, there is an error within SNAP though related to terrain flattening operation:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
org.esa.snap.core.gpf.graph.GraphException: [NodeId: Terrain-Flattening] Operator 'TerrainFlatteningOp': Unknown element 'reGridMethod'
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:85)
at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:58)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
at install4j.org.esa.snap.runtime.Launcher1159904018.main(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: Operator 'TerrainFlatteningOp': Unknown element 'reGridMethod'
at org.esa.snap.core.gpf.internal.OperatorContext.injectConfiguration(OperatorContext.java:1149)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:526)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
... 29 more
Caused by: com.bc.ceres.binding.ConversionException: Unknown element 'reGridMethod'
at com.bc.ceres.binding.dom.DefaultDomConverter.convertDomChildToPropertySet(DefaultDomConverter.java:175)
at com.bc.ceres.binding.dom.DefaultDomConverter.convertDomToPropertySet(DefaultDomConverter.java:165)
at com.bc.ceres.binding.dom.DefaultDomConverter.convertDomToValue(DefaultDomConverter.java:158)
at org.esa.snap.core.gpf.internal.OperatorContext.configureOperator(OperatorContext.java:1165)
at org.esa.snap.core.gpf.internal.OperatorContext.injectConfiguration(OperatorContext.java:1145)
... 33 more

Error: [NodeId: Terrain-Flattening] Operator 'TerrainFlatteningOp': Unknown element 'reGridMethod'
UTM WGS84
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file /working/S1A_20200905_175742_175807/output/S1A_05Sep2020_175742_175807/VV/GEO/S1A_05Sep2020_175742_1
75807_VV_Sigma0_APGB_UTMWGS84_TC.tif
SEVERE: com.sun.xml.bind.v2.util.XmlFactory: null
org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
at org.apache.xerces.parsers.AbstractSAXParser.setFeature(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.setFeatures(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParserImpl(Unknown Source)
at org.apache.xerces.jaxp.SAXParserFactoryImpl.setFeature(Unknown Source)
at com.sun.xml.bind.v2.util.XmlFactory.createParserFactory(XmlFactory.java:106)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.getXMLReader(UnmarshallerImpl.java:124)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:214)
at it.geosolutions.imageioimpl.plugins.tiff.gdal.GDALMetadataParser.parse(GDALMetadataParser.java:60)
at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.initializeFromMetadata(TIFFImageReader.java:1337)
at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.seekToImage(TIFFImageReader.java:865)
at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.seekToImage(TIFFImageReader.java:822)
at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.getWidth(TIFFImageReader.java:1080)
at org.esa.snap.dataio.geotiff.GeoTiffImageReader.getImageWidth(GeoTiffImageReader.java:160)
at org.esa.snap.dataio.geotiff.GeoTiffProductReader.readProduct(GeoTiffProductReader.java:230)
at org.esa.snap.dataio.geotiff.GeoTiffProductReader.readProductNodesImpl(GeoTiffProductReader.java:136)
at org.esa.snap.core.dataio.AbstractProductReader.readProductNodes(AbstractProductReader.java:178)
at org.esa.snap.engine_utilities.gpf.CommonReaders.read(CommonReaders.java:80)
at org.esa.snap.engine_utilities.gpf.CommonReaders.readCommonProductReader(CommonReaders.java:59)
at org.esa.snap.engine_utilities.gpf.CommonReaders.readProduct(CommonReaders.java:33)
at org.esa.snap.dem.dataio.FileElevationModel.init(FileElevationModel.java:53)
at org.esa.snap.dem.dataio.FileElevationModel.<init>(FileElevationModel.java:49)
at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.getElevationModel(RangeDopplerGeocodingOp.java:483)
at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.computeTileStack(RangeDopplerGeocodingOp.java:843)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:116)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)

INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.

Has anyone else observed this issue, or can anyone suggest something else I can try please?

Thanks in advance for any assistance.