Is pyroSAR a good substitute for python based SNAP toolbox solution?

Microwave Toolbox
#3

Hi John,

Thanks for the prompt response. Yeah, I am trying to create a fully python based workflow for my project which has the Sentinel-1 pre-processing part at the moment in SNAP toolbox. Moving it out and working with pyroSAR would definitely help to create a consistent solution.

I however, did not find the batch processing part in the pyroSAR documentation. Can you point it out specifically where it is or the class name I need to look at?

Thanks
Sid

#4

Hi Sid,
to my understanding batch processing is basically just processing a list of scenes with a single workflow. So in SNAP’s graph builder one would create a workflow XML file and load it into the batch processing module together with the list of scenes to then process them in sequence.
pyroSAR has a little different approach in that it creates an individual workflow file for each scene, processes the scene and then moves to the next one. The idea is to treat these XML files as additional metadata attached to the processing result so that in the future it is always clear how, i.e. with which nodes and their parameters, a particular scene was processed.
So basically this is just a loop iterating over a list of scenes, which is this part in the docs:

for scene in selection_proc:
    geocode(infile=scene, outdir=outdir, tr=resolution, scaling='db', shapefile=site)

The function geocode first writes an XML file individual to the scene, i.e. same naming convention as the output GeoTiffs and the input and output products hard-coded in the file, and then passes the file to the GPT command line tool. The output might look something like this:

S1A__IW___A_20180829T170656_tnr_bnr_Orb_Cal_ML_TF_TC_dB_proc.xml
S1A__IW___A_20180829T170656_VH_tnr_bnr_Orb_Cal_ML_TF_TC_dB.tif
S1A__IW___A_20180829T170656_VV_tnr_bnr_Orb_Cal_ML_TF_TC_dB.tif

The naming scheme is consistent for all sensors and products containing some basic scene metadata and the processing steps.
In its basic configuration, only the workflow and one GeoTiff for each polarization are saved. Ancillary products, like the local incident angle map, can be exported with parameter export_extra.

Cheers,
John

3 Likes
#5

Hi John,

Thanks for the update on batch processing. This clears the batch process with pyroSAR. I am currently writing the code for enabling the batch script, will update you once its done.

Sid

1 Like
#6

FYI…pyroSAR version 0.11 is now available via PyPI and Anaconda.

4 Likes
#7

Hi @johntruckenbrodt,

It’s super handy you’re around on the forums, I found pyroSAR a while back and was keen to know more!

Could you possibly let me know if there’s a benefit to me switching from just calling gpt (via subprocesses) to using pyroSAR?

The processing I’m currently doing to imagery is Read > Radiometric Calibration > Terrain Correction > Write (For Sentinel 1) - The gpt workflow is quite trivial and I’m wondering if I’d gain much benefit using your library? (I’d love to if it’l benefit me!)

Cheers and awesome work!

#8

Hi @CiaranEvans,

sure thing, the forum is really the place to be for us SAR enthusiasts :wink:.
Whether you can benefit from pyroSAR of course largely depends on your needs and the way you’d like to work with the data. Calling GPT via subprocesses sure works fine, pyroSAR also does that in the background.
However, pyroSAR goes further so that the whole data processing pipeline becomes easier to use. There are several things to it, but for now I’ll stick to those that might be relevant for your use case, i.e. Sentinel-1 GRD processing with SNAP. I assume that you have the S1 raw data storage figured out and have a working processing chain so I won’t go into raw data management. The question is, how flexible do you need your workflow to be in the long run?
For example, do you always process the whole scene or are you sometimes only interested in a small area? pyroSAR makes it quite easy to load your existing workflow and insert a Subset node with coordinates from some bounding box. Or are you interested in ancillary products like the local incident angle map from time to time and need to output them as well? These kinds of modifications to existing workflows are quite easy to do on the fly with pyroSAR’s SNAP API.
On the performance side, the ability to split workflows can significantly speed up processing. Instead of executing the whole workflow at once, pyroSAR can split it and execute the sub-workflows in sequence. In your case this could look like this:

Read > Radiometric Calibration > Write
Read > Terrain Correction > Write

This way the processing is faster and needs less memory. I agree though that this should not be necessary and SNAP will eventually be just as efficient for long workflows as it is for short ones.
On top of that, I occasionally discovered some output formatting issues. For example, SNAP does not properly write the nodata values to the GeoTiff files. Also, I found that the pixel ordering in the GeoTiffs is quite unusual and cumbersome to read in other software. I had addressed this here. Those issues might be fixed by now but sure saved my day some time ago.

One feature that really gets me excited lately is the ability to directly connect it to my analysis environment. I do a lot of time series analyses and want to treat all processed images of an area as one 3D cube. For this I am developing spatialist, which is pyroSAR’s spatial data handling backend. Here’s a small example:

from pyroSAR.ancillary import find_datasets, seconds
from spatialist import Raster

# some directory containing processed data
directory = 'E:\\DATA'

# collect all files with the pyroSAR naming pattern
files = find_datasets(directory, polarization='VV')

# sort the files by their acquisition time
files = sorted(files, key=seconds)

# connect all files as a 3D object and read a subset into a numpy array
with Raster(files)[0:100, 0:100, :] as ras:
    arr = ras.array()

Here pyroSAR’s consistent naming scheme comes into play making it easy to keep track of processed scenes. However, this really excels when using data from different sensors.

Okay, that’s already quite a long message so I’ll leave it here. I hope this sheds some light and you are still interested in using it. I’ll be around to answer more questions of course.

Cheers,
John

2 Likes
#9

Thanks John!

#10

Awesome, thanks for such a great reply!

It’s interesting to see what you’ve done with pyroSAR, also somewhat difficult to decide whether I will need more flexibility.

I didn’t think about running the Calibration and Correction separately though, that’s an interesting point that it is quicker. I will have to investigate that!

#11

no worries mate!
In German we have a saying “Mit Kanonen auf Spatzen schießen”. No point in “shooting sparrows with cannons”. Big tools might come with limitations.
Splitting might not make much of a difference in your case as it is quite a short workflow. It sure does with longer workflows as applied by pyroSAR.snap.geocode:

1448×2956

(See the link above for image description)

3 Likes
#12

What a stellar saying! Definitely appreciate that benefit!

I know it’s not related to pyroSAR (but as you have a way of processing using SNAP in a different way to me) - have you by any chance seen Graph Builder Subset and Terrain Correction Error Sentinel 1? I assume the problem I’m having would persist unless your subsetting is something new and fancy?

#13

Mh, no idea. I directly insert the WKT into the XML and it never failed me. Maybe this workflow helps to figure it out. It’s an older one though and might not work out of the box.
S1A__IW___D_20180826T053453_tnr_bnr_Orb_Cal_ML_TF_TC_dB_proc.xml (8.7 KB)

#14

Hey everyone,
I just came across this post and was wondering if it might be possible to put this workflow together with pyroSAR. I’m not quite sure after looking into the documentation, can anyone help?

SNAP steps:

  1. Read
  2. TOPSAR Split
  3. Apply Orbit file
  4. Back Geocoding
  5. Enhanced spectral diversity
  6. Coherence
  7. Deburst
  8. Multilook
  9. Terrain Correction
  10. Subset
  11. Write

Thanks in advance!

#15

Hi @mille1991,
sorry for the late response, I was on leave for some months.
There is no ready recipe available in pyroSAR for this kind of workflow, but it should be quite easy to put together using pyroSAR’s SNAP API. This way you can easily create an XML workflow file and then run it in SNAP either via the GUI or the command line tool. Please don’t hesitate to ask further questions here or on GitHub.
Cheers

1 Like
#17

Hi @mahdi_sbr77,
first it is important to consider what benefit you expect from having a 4-band image. The images from pyroSAR are exported as single band grayscale GeoTiffs, one for each polarization. This is in my opinion much easier to handle because one can immediately see what information is available for a scene.
Extending these files to four bands does only increase the file size and I can see no benefit in that. The way I see it, this export functionality of SNAP is intended for producing quicklook images which do not contain scientific measurements. In this case a grayscale png would make much more sense.
I just checked this export myself and the R, G and B bands contain exactly the same information like expected. If you want some colored image you can for example use VV for red, VH for green and their ratio VV/VH for blue or any other combination you seem fit. If you want to do this in Python you might want to explore the GDAL library, this is however beyond the scope of pyroSAR.
Cheers

#19

There is a function png in pyroSAR’s accompanying package spatialist that might be of help. It does obviously not offer the export to GeoTiff but the code might help you find what you are looking for.

#22

For this I suggest you search other topics in the STEP forum. I am sure this has been described before and it would not make sense to double it here. Otherwise you might want to refer to the gpt documentation. All the relevant links to the pyroSAR functions executing gpt are contained in the SNAP API documentation linked above.

1 Like
#24

There seems to be a problem with downloading the DEM, which causes SNAP to infinitely try to download the missing DEM tiles. You might wanna try adjusting the URL for 3 arcsec SRTMs in the /etc/snap.auxdata.properties file of your SNAP installation. This one should work:
https://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF
I have just added a new section troubleshooting in the SNAP API documentation explaining how these kind of errors are easy to identify using your case as example.

1 Like
#25

Dear Johntruckenbrodt,

I’m testing your nice module, I’m reading the last document has been published 09-March-2021,

My first question is,

Is there any separate object or method available to implement the flowchart you mentioned, step by step, not all the steps at once?,

I applied

from pyroSAR.snap import geocode
geocode(scene,
outdir=‘D:\pyrosar_test\orb’,
allow_RES_OSV=True)

It works fine

The following code,

from pyroSAR import Archive
dbfile = ‘D:\pyrosar_test\scenes.db’
with Archive(dbfile) as archive:
archive.insert(scene)

gives the following error,

image

Also the following code,

from pyroSAR.snap.auxil import gpt
gpt(‘D:\pyrosar_test\spdivpy201524.xml’, ‘D:\pyrosar_test’)

gives the following error,

image
image853×61 4.74 KB

The platform is python 3.6. and the OS, is windows 10. by the way I’m using spyder V. 4.1.3

#26

hI @falahfakhri. I am not quite sure what you mean by implementing flowcharts. If you are referring to creating SNAP workflows you might want to check out pyroSAR’s SNAP API. With it you can create arbitrary SNAP workflows. the geocode function just is a complex example of how to use this API.
Your error message hints to missing dependencies, libspatialite in this case. This module is needed for creating SQLite databases with spatial functionality. If you are running Windows you best install anaconda/miniconda and install pyroSAR into a conda environment. All dependencies, including libspatialite, will be installed with it.

#27

Thanks for your response,

First would you please to have a look again at my post the end of the post.

Second I meant something like,

“from pysar import ApplyOrbit”

“ApplyOrbit(scene)”

“Sigma0Cal(scene)”

Something like this, because I think you’re offering a tool to process radar data such as S-1 GRD, Therefore gives the users to work in flexible way according to their need, for instance, if I create an arbitrary workflows in SNAP, the gpt and bash are very convenient to me out of your module,

what is the new thing could get from your module?

I think I still prefer to work in python 3.6 for many other works, I think it’s stable and better than anaconda or miniconda in many aspects.