Sentinel1 multi-temporal coherence fails on version 10

alattaruolo · September 10, 2024, 8:41am

Hello, I am trying to run this graph

that worked on version 9.0.0 and now in version 10.0.0 is not working.

This is the error that I get:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.esa.snap.runtime.Engine (file:/home/mithra/esa-snap/snap/modules/ext/org.esa.snap.snap-core/org-esa-snap/snap-runtime.jar) to method java.lang.ClassLoader.initializePath(java.lang.String)
WARNING: Please consider reporting this to the maintainers of org.esa.snap.runtime.Engine
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: org.esa.snap.core.util.ServiceLoader: org.esa.snap.core.gpf.OperatorSpi: Provider eu.esa.opt.meris.sdr.aerosol.AerosolMergerOp$Spi not found
WARNING: org.esa.snap.core.util.ServiceLoader: org.esa.snap.core.gpf.OperatorSpi: Provider eu.esa.opt.meris.sdr.aerosol.ModisAerosolOp$Spi not found
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
-- org.jblas ERROR Couldn't load copied link file: java.lang.UnsatisfiedLinkError: /tmp/jblas7139374108146818383/libjblas.so: libgfortran.so.5: cannot open shared object file: No such file or directory.

WARNING: eu.esa.sar.insar.gpf.coregistration.CreateStackOp: Unable to calculate baselines. 'void org.jblas.NativeBlas.dgemm(char, char, int, int, int, double, double[], int, int, double[], int, int, double, double[], int, int)'
'void org.jblas.NativeBlas.dgemm(char, char, int, int, int, double, double[], int, int, double[], int, int, double, double[], int, int)'
 done.

Error: [NodeId: Coherence] 'void org.jblas.NativeBlas.dgemm(char, char, int, int, int, double, double[], int, int, double[], int, int, double, double[], int, int)'
org.esa.snap.core.gpf.graph.GraphException: [NodeId: Coherence] 'void org.jblas.NativeBlas.dgemm(char, char, int, int, int, double, double[], int, int, double[], int, int, double, double[], int, int)'

AccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
    at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
    at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
    at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
    at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:71)
    at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: 'void org.jblas.NativeBlas.dgemm(char, char, int, int, int, double, double[], int, int, double[], int, int, double, double[], int, int)'
    at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:434)
    at eu.esa.sar.insar.gpf.CoherenceOp.initialize(CoherenceOp.java:232)
    at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
    at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
    at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:359)
    at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
    ... 31 more
Caused by: java.lang.UnsatisfiedLinkError: 'void org.jblas.NativeBlas.dgemm(char, char, int, int, int, double, double[], int, int, double[], int, int, double, double[], int, int)'
    at org.jblas.NativeBlas.dgemm(Native Method)
    at org.jblas.SimpleBlas.gemm(SimpleBlas.java:247)
    at org.jblas.DoubleMatrix.mmuli(DoubleMatrix.java:1793)
    at org.jblas.DoubleMatrix.mmul(DoubleMatrix.java:3166)
    at org.jlinda.core.utils.PolyUtils.polyFit(PolyUtils.java:159)
    at org.jlinda.core.utils.PolyUtils.polyFitNormalized(PolyUtils.java:62)
    at org.jlinda.core.Orbit.computeCoefficients(Orbit.java:118)
    at org.jlinda.core.Orbit.<init>(Orbit.java:95)
    at eu.esa.sar.insar.gpf.CoherenceOp.metaMapPut(CoherenceOp.java:306)
    at eu.esa.sar.insar.gpf.CoherenceOp.constructSourceMetadata(CoherenceOp.java:274)
    at eu.esa.sar.insar.gpf.CoherenceOp.initialize(CoherenceOp.java:212)
    ... 35 more
-- org.jblas INFO Deleting /tmp/jblas7139374108146818383/libgfortran-4.so
-- org.jblas INFO Deleting /tmp/jblas7139374108146818383/libjblas_arch_flavor.so
-- org.jblas INFO Deleting /tmp/jblas7139374108146818383/libjblas.so
-- org.jblas INFO Deleting /tmp/jblas7139374108146818383/libquadmath-0.so
-- org.jblas INFO Deleting /tmp/jblas7139374108146818383

Apparently the microwave toolbox has problems reading the data…

diana_harosa · September 10, 2024, 12:19pm

@jun_lu can you help?

jun_lu · September 10, 2024, 2:01pm

Could you please provide more details regarding your processing (e.g. the source products and the parameters used for each operator) so that we can reproduce the error? Thank you

alattaruolo · September 10, 2024, 2:06pm

I used the following files:

first file:
S1A_IW_SLC__1SDV_20231201T170659_20231201T170726_051464_06361B_107C.zip

second file:

S1A_IW_SLC__1SDV_20231213T170659_20231213T170726_051639_063C47_9EFE.zip

Then the following parameters:

-Psubswath=IW2 -Pburst1=7 -Pburst2=7

Then the whole command:

gpt assets/s1coherence.xml -Pimage1_fpath=S1A_IW_SLC__1SDV_20231201T170659_20231201T170726_051464_06361B_107C.zip -Pimage2_fpath=S1A_IW_SLC__1SDV_20231213T170659_20231213T170726_051639_063C47_9EFE.zip -Psubswath=IW2 -Pburst1=7 -Pburst2=7 -Pcoherence_fpath=20231201T17065__20231213T17065_IW2_B77_MTC.tif

the coherence processing graph:

s1coherence.xml (7.3 KB)

MBG · September 11, 2024, 6:04am

By the way, why are you using the complete scenes for a single burst processing? Sentinel-1 burst downloads are available from the Alaska Satellite Facility (ASF) Datastore, making the process much more efficient.

alattaruolo · September 11, 2024, 9:27am

By the way, why are you using the complete scenes for a single burst processing? Sentinel-1 burst downloads are available from the Alaska Satellite Facility (ASF) Datastore, making the process much more efficient.

It is still possible to do it through snap using the gpt command? If so can you provide an example of the command execution? Thanks in advance, I am not an expert on this matter I am just executing the code that is generated from a script and reported this error since it was working on the version 9.0.0.

MBG · September 11, 2024, 9:44am

Potentially, yes, you just need to download the bursts by their identifiers and convert them into the common SLC scene format. I don’t use SNAP because it’s quite slow and lacks many key InSAR features. However, you can find a correlation calculation example in the interactive Google Colab notebook titled ‘Flooding [Correlation] Map: Kalkarindji, NT Australia, 2024’ at https://insar.dev. Just click on the example to open it and run the processing with a single click. You can download the results or run the process locally on Windows, macOS, Linux, etc. Replace the specified SCENES with your own scenes, subswaths, or bursts, and define a crop area if you prefer, then run the process to get the result.

alattaruolo · September 11, 2024, 9:46am

However, you can find a correlation calculation example in the interactive Google Colab notebook titled ‘Flooding [Correlation] Map: Kalkarindji, NT Australia, 2024’ at https://insar.dev .

Perfect thanks I will try to have a look on it.

MBG · September 11, 2024, 11:36am

By the way, for your scenes (including all bursts for subswath 2), it generates the correlation map shown below:

alattaruolo · September 11, 2024, 11:46am

Thank you very helpful, then now we just wait to see if there are problems in version 10.0.0 of snap.

jun_lu · September 20, 2024, 7:47pm

We have processed in SNAP 10 the two products above using the graph that you provided. Unfortunately we cannot reproduce the error that you had encountered. Here are the result:

Detail parameters used in the processing can be found in the attached graph.
s1coherence (1).xml (7.8 KB)

mzanetti · September 23, 2024, 7:44am

Hi @jun_lu, did you run it from the GUI or from command line as we did? (I work with @alattaruolo on the same files). Thanks

jun_lu · September 23, 2024, 4:21pm

I first ran it from the GUI and then ran it from the command line, and did not have errors in both cases. My command line is as “gpt s1coherence.xml” because all the processing parameters are saved in the graph. You can try to generate the graph in the Graph Builder with all the desired parameters and save it. Then run it in the command line to see if that makes difference.

mzanetti · September 24, 2024, 10:00am

@jun_lu I’ve tried myself with SNAP version 10.0.0 and it works, on Windows. I therefore believe the problem of @alattaruolo is related to the Ubuntu version. Is it possible that there is a bug only in the Ubuntu version?
Thanks

mzanetti · September 27, 2024, 12:16pm

Hi @MBG I’ve managed to download bursts from the ASF. Thank you very nice improvement. However, I cannot run my graph for Coherence calculation on SNAP. Can you help me? Using the same graph shared by @alattaruolo I get an error from the TOPSplit saying that the input (the Burst image from ASF) is not a SAR product. Can you help? Thanks

MBG · September 27, 2024, 12:34pm

We have no involvement with SNAP due to data format issues and its high resource consumption and so on. I’ve developed the open-source PyGMTSAR (Python InSAR) library, which is available on Google Colab, through Docker images, and more. PyGMTSAR can perform processing much faster, even on systems with only a few gigabytes of RAM, and it provides 2D, 3D, and web map outputs. The PyGMTSAR notebook shared above produces results specific to your case. I’m not sure if the same outcome can be replicated using SNAP.

mzanetti · September 27, 2024, 12:40pm

Sorry, I am not getting it. What do you mean you have no involvement with SNAP? Isn’t this SNAP forum?

MBG · September 27, 2024, 12:50pm

NASA’s ASF datastore provides access to individual Sentinel-1 bursts, and my InSAR software includes a downloader and preprocessor specifically for them. However, I am not affiliated with the SNAP development team, so I can’t offer support if SNAP is unable to process these bursts. They work well for NASA ASF team and in my InSAR package. Actually, the downloaded bursts are saved as standalone, unpacked Sentinel-1 single-burst scenes, each containing a single GeoTIFF and an accompanying XML file. If you don’t have additional information, how can we proceed in finding a solution?

mzanetti · September 27, 2024, 12:53pm

Ok, and inf fact here I am asking in the SNAP forum if the Software is enabled to read these bursts products (thank you again for letting me know about them). I am not saying that SNAP cannot process them, I am actually asking IF it can process them

My need is to compute Coherence between two products, before I used to process two entire S1 images and debursting after that. How should I modify the graph (previous attachment) in order to do this?

Thanks

MBG · September 27, 2024, 12:57pm

As far as I know, SNAP can process them. It is recognized as bursts-compatible software by the NASA ASF team.