Python S1-TOPS-SPLIT Analyzer

panjibrotoisworo · March 14, 2021, 4:13am

Hi everyone,

I’m sharing a Python tool I made where you can visualize and export the Sentinel-1 S-1 TOPS SPLIT Data simply called S-1 TOPS SPLIT Analyzer. Simply input the path of the downloaded S1 image and choose which subswaths you want. Input image should still be a ZIP file. Then you can use the class methods to extract the data. This tool can be used in the command line and as an import.

Command line sample:
python stsa.py -zip S1_image.zip --swath iw2 iw3 -polar vv -shp out_shp.shp -csv out_csv.csv -json out_json.json

Python sample:

from stsa import TopsSplitAnalyzer

s1 = TopsSplitAnalyzer(image='S1_image.zip', target_subswaths=['iw1, iw2'], polarization='vh')
s1.to_shapefile('out_shp.shp')

Files and documentation are available here:

You can specify specific subswaths and polarizations to extract. You can export it as a shapefile, csv, or JSON. You can also view it on a webmap (You need Jupyter Notebook).

In addition, you can specify one custom shapefile to add to the map. For me this is useful because I can quickly view which subswaths and bursts I need to use for any type of work. In the image below, it shows the output of the webmap function and the custom shapefile is marked as red.

I welcome any feedback or issues. Hope it helps!

Sample output:

s1.visualize_webmap(polygon='aoi.shp')

Exported shapefile:

s1.to_shapefile('subswaths.shp')

EDIT: added codeblock and some instructions for clarity.

marpet · March 14, 2021, 9:21am

Thanks for sharing you tool.
I have moved your post into the dedicated category ‘Show Room’.

ABraun · March 14, 2021, 10:21am

thank you for sharing, this is a very useful tool!

I noticed two things:

In your documentation it should be

s1 = TopsSplitAnalyzer(‘S1_image.zip’, target_subswath=[‘iw1, iw2, iw3’])

instead of

s1 = TopsSplitAnalyzer(‘S1_image.zip’, target_subswaths=[‘iw1, iw2, iw3’])

When I run it on a loaded zip file (this works), I get this error:

Traceback (most recent call last):
File “”, line 1, in
s1.to_shapefile(‘test.shp’)

File “C:\Python36\lib\site-packages\topssplitanalyzer.py”, line 196, in to_shapefile
File “C:\Python36\lib\site-packages\topssplitanalyzer.py”, line 162, in _create_subswath_geometry
File “C:\Python36\lib\site-packages\topssplitanalyzer.py”, line 83, in _load_metadata

Exception: No XML metadata file found

Do you have an idea on this?

panjibrotoisworo · March 14, 2021, 10:54am

I pushed a new update to the Github repo. Thanks for reporting @ABraun. I missed this because I didn’t name the arguments in my testing . I renamed the input so that it matches the reference in the documentation target_subswaths. I also included more helpful error messages in that case.

No XML metadata found happened because it is trying to find the matching metadata file based on string matching. That error can occur if the input target_subswaths and polarization are not typed correctly.

Also I tested this with SLC data only.

ABraun · March 14, 2021, 11:15am

thank you for the update, now I receive
Loaded ZIP file: S1A_IW_SLC__1SDV_20201123T142458_20201123T142525_035377_042241_C054.zip
Found 6 XML paths

Sounds better.

But upon s1 = TopsSplitAnalyzer(slc, target_subswaths=['iw1, iw2, iw3'], polarization='vv')

I receive
Exception: Found no matching XML file with target subswath "iw1, iw2, iw3" and target polarization "vv"

Maybe I am missing a something here…

panjibrotoisworo · March 14, 2021, 11:19am

target_subswaths should be a list of strings, not one string inside a list.

This:

target_subswaths=['iw1, iw2, iw3']

Should be this:

target_subswaths=['iw1', 'iw2', 'iw3']

ABraun · March 14, 2021, 11:20am

thank you - I am getting closer. But something is still raising an error.

s1.to_shapefile(‘D:\Geodaten\S1\Dubai_SLC\data.shp’)
Loaded location grid with 9 bursts
Traceback (most recent call last):

File “”, line 1, in
s1.to_shapefile(‘D:\Geodaten\S1\Dubai_SLC\data.shp’)

File “C:\Python36\lib\site-packages\topssplitanalyzer.py”, line 209, in to_shapefile

File “C:\Python36\lib\site-packages\topssplitanalyzer.py”, line 177, in _create_subswath_geometry

File “C:\Python36\lib\site-packages\topssplitanalyzer.py”, line 152, in _parse_subswath_geometry

File “C:\Python36\lib\site-packages\shapely\geometry\polygon.py”, line 240, in init
ret = geos_polygon_from_py(shell, holes)

File “C:\Python36\lib\site-packages\shapely\geometry\polygon.py”, line 494, in geos_polygon_from_py
ret = geos_linearring_from_py(shell)

File “shapely/speedups/_speedups.pyx”, line 319, in shapely.speedups._speedups.geos_linearring_from_py

TypeError: object of type ‘Point’ has no len()

panjibrotoisworo · March 14, 2021, 11:34am

Can you show print(s1.df)? The shapefile is stored in a geopandas dataframe and I use that to export the data. I think that the XML data is not being read correctly in your case but I need to see the dataframe to be sure.

ABraun · March 14, 2021, 11:37am

print(s1.df)
returns an empty object (“None”)

Should the shapefile exist before executing the script?

panjibrotoisworo · March 14, 2021, 11:41am

In my case the script still works if a shapefile exists or not.

Can you share the XML data from your image so I may test it? Inside the ZIP file they should be located in:

<your_image>.SAFE/annotation/<xml_data>

I don’t need the data in the calibration folder.

ABraun · March 14, 2021, 11:58am

sure

s1a-iw3-slc-vh-20201123t142459-20201123t142524-035377-042241-003.xml (771.4 KB)
s1a-iw3-slc-vv-20201123t142459-20201123t142524-035377-042241-006.xml (771.4 KB)
s1a-iw1-slc-vh-20201123t142500-20201123t142525-035377-042241-001.xml (864.1 KB)
s1a-iw1-slc-vv-20201123t142500-20201123t142525-035377-042241-004.xml (864.2 KB)
s1a-iw2-slc-vh-20201123t142458-20201123t142524-035377-042241-002.xml (856.5 KB)
s1a-iw2-slc-vv-20201123t142458-20201123t142524-035377-042241-005.xml (856.5 KB)

panjibrotoisworo · March 14, 2021, 3:53pm

The repo is updated.

@ABraun hopefully I figured out what was wrong. I think I am using a newer version of shapely which is more flexible. But according to this StackOverflow post, I was not creating the Polygon object correctly. I’ve fixed the code and followed the format given in the StackOverflow post so that it is compatible with older shapely versions. The code still works for me, but hopefully it works for you too.

ABraun · March 14, 2021, 6:58pm

thank you!
The CSV export works now - perfect

Does the export to GeoJSON need any arguments?

The shp is missing some coordinate transformation, but I can’t tell if this is related to the version of my packages or if it’s a general issue.

s1.to_shapefile(‘D:\Geodaten\S1\Dubai_SLC\mask.shp’)
Traceback (most recent call last):

File “”, line 1, in
s1.to_shapefile(‘D:\Geodaten\S1\Dubai_SLC\mask.shp’)

File “C:\Python36\lib\site-packages\topssplitanalyzer.py”, line 214, in to_shapefile
self.df.to_file(filename=output_file)

File “C:\Python36\lib\site-packages\geopandas\geodataframe.py”, line 504, in to_file
to_file(self, filename, driver, schema, **kwargs)

File “C:\Python36\lib\site-packages\geopandas\io\file.py”, line 128, in to_file
filename, “w”, driver=driver, crs=df.crs, schema=schema, **kwargs

File “C:\Python36\lib\site-packages\fiona\env.py”, line 400, in wrapper
return f(*args, **kwargs)

File “C:\Python36\lib\site-packages\fiona_init_.py”, line 277, in open
**kwargs)

File “C:\Python36\lib\site-packages\fiona\collection.py”, line 153, in init
self._crs_wkt = crs_to_wkt(crs_wkt or crs)

File “fiona_crs.pyx”, line 78, in fiona._crs.crs_to_wkt

CRSError: Invalid input to create CRS: EPSG:4326

cristianoLopes · March 14, 2021, 11:30pm

Very interesting indeed…

Some notes:

there’s a license discrepancy between the header of the “.py” file and the repo. One says GPL v3 the other AGPL v3. Any reason to choose GPL (didn’t check embedded dependencies) over any of the more recent licenses (Apache2, MPL2, etc)
You could/should add a requirements file to clearly indicate which dependencies are needed to use this.
You could/should use an “if name==main” code block with command line parsing so that people can use this as a command line tool…
Perhaps you can consider to use the OData API from the Open Hub and try to obtain the product nodes that you need directly from there, thus avoiding to download the large product zip file.

cristianoLopes · March 14, 2021, 11:31pm

perhaps this is more suitable in meta… there isn’t any item from SNAP within this code…

panjibrotoisworo · March 15, 2021, 1:09am

@ABraun Glad to hear it sort of works. The error I think it might be related to your versions. May I know your Geopandas and Shapely version? This can be done by typing <package>.__version.__. But I think I’ll create a more proper installation process based on the advice on @cristianoLopes.

@cristianoLopes.

I admit I’m not very familiar with licenses. I’m only familiar with GNU GPL and that it allows for free distribution but if used in a commercial environment they need to keep the code open-source. Can you suggest a more suitable license?
I’ll look into creating a more proper installation process to avoid the errors seen above
That’s a good idea. I’ll add command line functionality to the code.
I’ll explore the OData API that looks interesting. I think this can allow people to preview the data without downloading the full image.

Sorry for the initial trouble but I appreciate the advice and feedback so far. I will update this soon

panjibrotoisworo · March 15, 2021, 6:21am

Hi everyone the repo is updated. Hopefully this fixes the issues.

Revised dependencies for better stability. Please update or reinstall dependencies if they are in your environment. If errors persist, try creating a new virtual environment. Tested on Python versions 3.6+ (Test results)
Revised JSON output. Now an output file can be specified and is properly formatted
Added command line function
Renamed main file to stsa.py to make it easier to type
Updated README for better clarity and includes command line documentation

marpet · March 15, 2021, 8:04am

At the moment the category description says:

" In this Show Room category, you can present the results you have achieved with or without SNAP and related tools.
You can show interesting processing results, papers, SNAP plugins and other similar things."

So, according to the description it fits into the category.
But actually this discussion is meta

cristianoLopes · March 15, 2021, 8:50am

sorry - being a bit exaggerate here… I kind of formed a wrong misconception of where things should go and didn’t bother to read the current category descriptions.

In essence I my trying to understand how to best organise things so that it is easy(ier) to distinguish between SNAP related discussion, SNAP Plugin/community tools, Sentinel services and tools and others… probably finish this in private.

cristianoLopes · March 15, 2021, 9:21am

Cool update… you are really committed to this.

Let’s see the problem I highlighted is that right now it is not clear if you are using GPL3 or AGPL3 you have both. Have a look at something like TL;DR Legal to get a better understanding of the different licenses.
What GPL does is that it forces anyone creating a derived product that uses yours (in any different ways) to also license that software as GPL - in a sense it is viral. There are several variations of GPL, GPL itself. AGPL which says that if you use the item in a service also all the code for the service must be licensed as AGPL. LGPL that says that if you use the code as a external library than you can decide on your own license.
Many organisations will not even look at a GPL licensed software because of its viral nature, even if they just want to use it (i.e. viral nature would not impact them).
I suggested Apache license 2 or Mozilla Public License 2…