Automated script that produces the necessary files for SBAS in MintPy

Dear community and Linux users,

I have been working on projects lately regarding ground displacement using SBAS methodology in MintPy. As we know, processing all the Sentinel-1 scenes for MintPy is very tedious and time consuming process. Hence, for my own benefit, i tried to automate the InSAR processing using a bash script. The script works in Linux environment.

Here, i present you my bash script where the InSAR processing can be executed in an automated way.

How to run the script

Let’s assume we have the following directory structure.

my_project
│   InSAR_processing_for_mintpy.sh
│   
└───raw
│   │  S1A_IW_SLC_20210105_20210210.zip
│   │  S1A_IW_SLC_20210105_20210310.zip
|   |  S1A_IW_SLC_20210105_20210210.zip
│   
└───output
│      
└───AOI
|
└───graphs
cd my_project
bash InSAR_processing_for_mintpy.sh graphs/InSAR_processing.xml raw aoi/aoi.txt output/ IW1

  • InSAR_processing.xml is the graph that does the main InSAR pre-processing steps in the following order:
    - Reads both scenes
    - Performs TOPSAR-split
    - Apply orbit files
    - Performs Back-geocoding
    - Performs Enhanced-Spectral-Diversity
    - Interferogram creation
    - Performs TOPSAR-Deburst
    - Subset the scenes
    - Performs TopoPhaseRemoval
    - Performs GoldsteinPhaseFiltering
    - Write the product

If someone wants to include further steps into the processing chain (e.g multi-looking), the user can include the multi-looking parameters into the InSAR_processing.xml graph.

  • raw is the folder where all the raw Sentinel-1 SLC data are located. The user can specify a different name where he/she can keep the raw data for processing. Let’s say we want to create a netwrok of 3 intererograms. We load the 4 scenes into the raw folder (e.g scene1, scene2, scene3, scene4) the following itnterferograms will be created automatically (and all the products for MintPy) scene1 with scene2, scene1 with scene3, scene1 with scene4. Then, the user repeats the process with the next 4 scenes till all raw data are processed

  • aoi.txt is used to subset the S-1 scene. The location of the .txt file should be specified on the command line as shown above. This file must contains a WKT in geographic coordinates (WGS84). Those coordinates represent the AOI. The Wicket tool can be used to draw a polygon around the AOI and it provides the coordinates where we can copy and paste them into our .txt file

  • output is the directory where all the products that are produced from SNAP for MintPy are going to be written. The name of this folder MUST be named output. It is hardcoded in the bash script. The script also produces connected components whichs allows for phase correction in MintPy

  • IWI is the S-1 sub-swath. The user needs to check in which sub-swath the AOI falls in (e.g IW1, IW2 or IW3) and prodive it to the command line

Important notes

  • In the bash script ( InSAR_processing_for_mintpy.sh ) the user must change the first line regarding the gptPath location in the machine. In my case the GPT is located under /home/io/snap/bin/gpt . It wil be different for other users
  • This script was only tested on my laptop using SNAP version 8. If different versions of SNAP are used, I do not promise that the script will run smoothly
  • At the very end of the script, you will see some lines of code commented. If the user wants to keep all the products that have been produced in each step for inspection, then keep the lines commented. If the user is only interested in the products that are used in MintPy, he/she can remove the comments. The script will permanently delete all the unnecessary products and will keep only the ones located in the folder interferorams ready to be used in MintPy. In that way the folder does not get cluttered with many different files
  • The script supports only Sentinel-1A products at the moment. If anyone tries to precess Sentinel-1B data it will not work. This is because the script uses lots of regular expressions and other things where it points everything to data regarding S1A

Things to improve

  • Adjust the script so that it can process Sentinel-1B data
  • Allow users to create an output directory with a different name other than the hardcoded output
  • Adjust the script so that it merges all sub-swaths and process them all.

The final results that can be used in MintPy can be found within the folder called interferograms. The structure of the results is shown below. The script also outputs a DEM which is required.

results

This is not a perfect script. It is developed by me for my own benefits to make my life easier. Hence, I decided that i can potentially make someone else’s life easier as well.

Feel free to try it out and let me know if you succeeded

InSAR_processing_for_mintpy.sh (11.0 KB)
graphs.zip (5.4 KB)

9 Likes

Have a nice day that’s a great work. I am trying to use Gamma prrocessed data in MintPy., however always getting an error while load_data step. Error ; cantnot reshape array of size 24709908 into shape (28802). Any valueable insights ? best regard

Hello,

I have not used Gamma software before. I cannot help you with that
It might worth looking at Mintpy group. There are users who used Gamma software for InSAR analysis and posts their questions on that group

Ioannis

1 Like

Thank You Johngan. Yes now I am following that group.

johngan

I have a question.
When installing snaphu with snap, I would like to tell InSAR_processing_for_mintpy.sh where snaphu is placed.
Which part should I write? please tell me.

johngan

I have a question.
I received an error when I executed the attached sh file. I suspect that the placement of snaphu may not be correct. Could you please advise on the appropriate location for snaphu? I am using Ubuntu 22 as the operating system and SNAP8 for processing. Currently, I have placed snaphu in the default directory:

$HOME/.snap/auxdata

#######################—Performs Phase Unwrapping—########################

InSAR_processing_for_mintpy.sh: 行 114: command not found

#######################—SnaphuImport Operator - Load Unwrapped Phase—########################

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters

INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.

Executing processing graph

done.

Error: [NodeId: Read] The ‘file’ parameter is not set

Hello Freed-thinks,

The error indicates that something is wrong with snaphu. The script cannot find snaphu.
Make sure that snaphu is installed.

  • Type in the terminal snaphu -h. If you get no output, it means it has not been installed properly.
  • If not installed, install snaphu from Stanford website
  • The default location after the installation should be under /usr/bin/snaphu

Hello johngan.

Thank you for your reply.
In my environment, snaphu is located at /usr/local/snap/bin/gpt.
Therefore, when I set gptPath=/usr/local/snap/bin/gpt, I was able to perform the calculation up to SnaphuExport.xml.
However, at the point of SnaphuImport.xml, an analysis error occurs, similar to what I posted the other day. Do I need to make any modifications to SnaphuImport.xml?

Hello johngan.

I’m encountering an error on line 115 of InSAR_processing_for_mintpy.sh. Originally, it is written as follows:

$run_snaphu -g UnwPhaseConn_ifg_VV_${date_source_product_1}${date_source_product_1}.snaphu.img

I believe the second ${date_source_product_1} should be replaced with _${date_source_product_2}. Should it be modified as follows?

$run_snaphu -g UnwPhaseConn_ifg_VV_${date_source_product_1}_${date_source_product_2}.snaphu.img

Additionally, during SnaphuExport, it seems that UnwPhaseConn_ifg_VV~.img won’t be generated. Would it be correct to use the following syntax?

$run_snaphu -g Phase_ifg_VV_${date_source_product_1}_${date_source_product_2}.snaphu.img

Or is the absence of UnwPhaseConn_ifg_VV_${date_source_product_1}${date_source_product_2}.snaphu.img causing the analysis to fail?

Regarding SnaphExport.xml, on line 7, there is ${source_file}, but when the command is executed, it returns:

Error: [NodeId: Read] Specified 'file' [${source_file}] does not exist.

Where should this be defined in InSAR_processing_for_mintpy.sh?

Thank you.

Hello johngan.

Is it possible that Snaphu-Unwrapping needs to be executed manually?