Sentinel-1 GRD preprocessing graph

FedericoF · September 11, 2019, 2:11pm

Standard workflow for the preprocessing of Sentinel-1 GRDH data.

The GPF graph executes a Sentinel-1 GRDH preprocessing workflow that consists of seven processing steps, applying a series of standard corrections:

Apply Orbit File
Thermal Noise Removal
Border Noise Removal
Calibration
Speckle filtering (optional)
Range Doppler Terrain Correction
Conversion to dB

Sentinel-1 GRD products can be spatially coregistered to Sentinel-2 MSI data grids, in order to promote the use of satellite virtual constellations by means of data fusion techniques. Optionally a speckle filtering can be applied to the input image.

Here are some examples of command line

Process to 20 m without speckle filtering
gpt S1_GRD_preprocessing.xml -Presolution=20 -Porigin=10 -Pfilter='None' -Pdem='SRTM 3Sec' -Pcrs='GEOGCS["WGS84(DD)", DATUM["WGS84", SPHEROID["WGS84", 6378137.0, 298.257223563]], PRIMEM["Greenwich", 0.0], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH]]' -Pinput=S1A_IW_GRDH_1SDV_20160228T051920_20160228T051956_010142_00EF52_AB5E.SAFE -Poutput=S1A_IW_GRDH_1SDV_20160228T051920_20160228T051956_010142_00EF52_AB5E.dim

Process to 10 m with ‘Refined Lee’ speckle filtering
gpt S1_GRD_preprocessing.xml -Presolution=10 -Porigin=5 -Pfilter='Refined Lee' -Pdem='SRTM 3Sec' -Pcrs='GEOGCS["WGS84(DD)", DATUM["WGS84", SPHEROID["WGS84", 6378137.0, 298.257223563]], PRIMEM["Greenwich", 0.0], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH]]' -Pinput=S1A_IW_GRDH_1SDV_20160228T051920_20160228T051956_010142_00EF52_AB5E.SAFE -Poutput=S1A_IW_GRDH_1SDV_20160228T051920_20160228T051956_010142_00EF52_AB5E.dim

Description of command line options:

-Pinput input Sentinel-1 GRD ‘.SAFE’ file
-Poutput output Sentinel-1 GRD preprocessed file (BEAM-DIMAP format)
-Presolution set target spatial resolution
-Porigin allows to snap output grid to Sentinel-2 data grid, should be set to half the size of spatial resolution set using the ‘-Presolution’ option
-Pfilter defines the speckle filter to be used (only supported speckle filters available in SNAP, ‘None’ does not apply any speckle filtering)
-Pdem set the DEM to be used for terrain correction

References

Filipponi, F. (2019). Sentinel-1 GRD Preprocessing Workflow. In Multidisciplinary Digital Publishing Institute Proceedings (Vol. 18, No. 1, p. 11).

GPF graph

S1_GRD_preprocessing.xml (6.7 KB)

irgan · August 5, 2020, 9:57am

Hi there,
so I’m preprocessing S1 and the problem I am having is that some of the images were not processed completely, meaning only a clipping of the image is processed.
I did this firstly with the SNAP GUI, than I tried with the GPT. Finally I tried processing a single image with the graph. All the time I get only the same clipping processed, the rest is black. I tried saving the output as .dim and as .geotiff formats, both the same.
I am having this problem with number of images, here’s a snapshot:

The graph I’m using is almost the same as yours. preproc_S1_v1.xml (5.0 KB)
Any suggestions how I could proceed?

FedericoF · August 5, 2020, 5:09pm

Dear Irina,
I am not able to identify the problem from the screenshot and the attached graph.
I see you are using a ‘region’ and a ‘polygon’ variable for the ‘Subset’ operator, and I am wondering if that step may give some problems. Have you tried to run the graph without the ‘Subset’ node?
Another issue may be the available RAM for the process.

Maybe other users can give you some other suggestions to solve the problem.

Regards,

Federico

irgan · August 6, 2020, 8:59am

Hi Frederico, many thanks for your reply.
The ‘region’ and ‘polygon’ variables in the Subset work fine, because I’ve used this operator in other graphs and it’s subsetting correctly.
I have processed data with this graph for the last one year for my AOI and again, some of the images are processed correctly giving the expected result, and others are clipped

I did all the steps separately and it turns out that the clipping occurs just after the calibration step.
Best,
I

rbavery · February 26, 2021, 7:05am

I get an error when importing this graph into graph builder using SNAP 8.0.0

Java.lang.NullPointerException

ABraun · February 26, 2021, 12:08pm

the disadvantage with graphs is that they obscure the point within the processing chain where something went wrong.

Can you please share the graph in here?

rbavery · March 1, 2021, 9:39pm

Yes thanks for the quick reply, I found it on this current forum post: Sentinel-1 GRD preprocessing graph

And the graph I’ve attached.S1_GRD_preprocessing.xml (6.7 KB)

FedericoF · March 1, 2021, 10:52pm

Dear @rbavery,

the processing graph was conceived to process Sentinel-1 GRD data using gpt from command line.
I have just tested the graph using SNAP8 gpt, and I could successfully run it.
Please try to execute the graph from the command line using one of the examples shown in the first post.

Federico

rbavery · March 1, 2021, 11:23pm

Thanks a bunch @federicoF

I tried this, using the same args but a different GRDH input file (snapg is aliased to the correct gpt since I have another program called gpt already installed on my Mac):

→ snapg /Users/rave/ms-sar/preprocessing/S1_GRD_preprocessing.xml -Presolution=30 -Porigin=5 -Pfilter='Refined Lee' -Pdem='SRTM 3Sec' -Pcrs='GEOGCS["WGS84(DD)", DATUM["WGS84", SPHEROID["WGS84", 6378137.0, 298.257223563]], PRIMEM["Greenwich", 0.0], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH]]' -Pinput=S1A_IW_GRDH_1SSV_20141007T223418_20141007T223443_002729_0030F5_BB2B.SAFE -Poutput=S1A_IW_GRDH_1SSV_20141007T223418_20141007T223443_002729_0030F5_BB2B.dim

I get this initial error. java.lang.NullPointerException. So it looks like my installation has something set up differently than yours. Any ideas? I appreciate the help!

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.2.1 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
SEVERE: org.esa.snap.core.dataio.ProductIO: Error attempting to read S1A_IW_GRDH_1SSV_20141007T223418_20141007T223443_002729_0030F5_BB2B.SAFE with plugin reader org.csa.rstb.io.rcm.RCMProductReaderPlugIn@59cda16e
java.lang.NullPointerException
	at org.csa.rstb.io.rcm.RCMProductReaderPlugIn.findMetadataFile(RCMProductReaderPlugIn.java:67)
	at org.csa.rstb.io.rcm.RCMProductReaderPlugIn.getDecodeQualification(RCMProductReaderPlugIn.java:57)
	at org.esa.snap.core.dataio.ProductIO.getProductReaderForInput(ProductIO.java:301)
	at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:163)
	at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
	at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
	at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:85)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:58)
	at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
	at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
	at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
	at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
	at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
	at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
	at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
	at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
	at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
	at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)

After that error it continues to run, failing with a timeout error to apply an orbit file (probably because it couldn’t open the Sentinel-1 .SAFE) and downloading a DEM.

INFO: org.hsqldb.persist.Logger: dataFileCache open start
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: ApplyOrbit ignoring error and continuing: org.apache.http.conn.HttpHostConnectException: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
version = 2.36
.INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_24_16.zip
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip
..INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_25_16.zip
.10%....20%....30%....40%....50%....60%....70%....80%....90% done.

If I try to run it again after the DEM has downloaded, it fails with the same java null pointer error at the reading step.

FedericoF · March 2, 2021, 12:22am

Dear @rbavery,

from your gpt log I can see errors related to the download of Sentinel-1 precise orbit. DEM is correctly downloaded, and you can also think about using a more recently available DEM (e.g. ‘Copernicus 30m Global DEM’).

From the ‘Apply-Orbit File Operator’ help:
For Sentinel-1, Restituted orbit files and Precise orbit files may be applied. Precise orbits are produced a few weeks after acquisition. Orbit files are automatically download from Array’s servers. If an orbit file is not found, you may try looking for it in https://qc.sentinel1.eo.esa.int/ and placing the downloaded file into the auxdata folder.

I have also received the same error message in the past, that could be either related to server unavailability:

or local firewall configuration:

The GDAL version related INFO message could be manually fixed from SNAP8 by setting ‘Use internal GDAL version’ in: Tools -> Options -> S2TBX -> GDAL Library Loader.

Finally, I do not know why you get the error related to ‘org.csa.rstb.io.rcm.RCMProductReaderPlugIn.findMetadataFile’. Have you tried to set the ‘manifest.safe’ file inside the ‘.SAFE’ input folder as ‘-Pinput’ argument?

Federico

rbavery · March 2, 2021, 12:51am

Thanks a bunch for the tips. When running the same command (with a different DEM) in this docker container I get rid of a lot of those errors, with only this orbit error file remaining and a new error where the dem doesn’t finish downloading … Killed. This dem download error only happened with the docker image so I’m going to look at why that is and maybe report the issue on the mundialis repo.

It looks like the site to download the orbit files is down https://qc.sentinel1.eo.esa.int/

So I might just need to wait for this server issue to be fixed?

# rave at Ryans-MacBook-Pro.local in ~/ms-sar/preprocessing on git:main ✖︎ [16:55:36]
→ docker run snap
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
INFO: org.esa.s1tbx.commons.io.ImageIOFile: Using FileCacheImageInputStream
INFO: org.hsqldb.persist.Logger: dataFileCache open start
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: ApplyOrbit ignoring error and continuing: org.apache.http.conn.HttpHostConnectException: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
version = 2.36
INFO: java.util.prefs.FileSystemPreferences$1: Created user preferences directory.
.Copernicus_DSM_COG_10_S18_00_W063_00_DEM.tif
Downloading https://copernicus-dem-30m.s3.eu-central-1.amazonaws.com/Copernicus_DSM_COG_10_S18_00_W063_00_DEM/Copernicus_DSM_COG_10_S18_00_W063_00_DEM.tif to fulfill search of area -18.0, -63.0 at specified resolution 30
Downloaded file
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip
Copernicus_DSM_COG_10_S18_00_W062_00_DEM.tif
Downloading https://copernicus-dem-30m.s3.eu-central-1.amazonaws.com/Copernicus_DSM_COG_10_S18_00_W062_00_DEM/Copernicus_DSM_COG_10_S18_00_W062_00_DEM.tif to fulfill search of area -18.0, -62.0 at specified resolution 30
Downloaded file
.Copernicus_DSM_COG_10_S18_00_W061_00_DEM.tif
Downloading https://copernicus-dem-30m.s3.eu-central-1.amazonaws.com/Copernicus_DSM_COG_10_S18_00_W061_00_DEM/Copernicus_DSM_COG_10_S18_00_W061_00_DEM.tif to fulfill search of area -18.0, -61.0 at specified resolution 30
.Downloaded file
Copernicus_DSM_COG_10_S18_00_W060_00_DEM.tif
Downloading https://copernicus-dem-30m.s3.eu-central-1.amazonaws.com/Copernicus_DSM_COG_10_S18_00_W060_00_DEM/Copernicus_DSM_COG_10_S18_00_W060_00_DEM.tif to fulfill search of area -18.0, -60.0 at specified resolution 30
.10%Downloaded file
.Killed