8.0.x, CentOS 7, jblas, static and dynamic libraries

I am attempting to run ESD in 8.0.x, and I receive the errors below similar to those reported in: 27386, 24522, and 12023 . My CentOS 7 system upgrades are up to date. I have libgfortran3, 4, and 5 (which contain libquadmath?). My libgfortran references GLIBC_(2.14, 2.17, and older). My libm.so.6 includes GLIBC_2.15, and older.

@jun_lu Solution: at least for Linux with GLIBC older than 2.23, remove snap/s1tbx/modules/ext/org.jlinda.jlinda-* and replace with versions from SNAP 7.0.x.
i.e.: /bin/ln -fs snap_7.0.x/s1tbx/modules/ext/org.jlinda.jlinda-* snap_8.0.x/s1tbx/modules/ext/
I believe the problem is that 8.0.x jblas.jar includes a static build of libquadmath, that references a dynamic libm.so.6 at version GLIBC_2.23, which I and other distros do not have.

Errors:
– org.jblas ERROR Couldn’t load copied link file: java.lang.UnsatisfiedLinkError: /tmp/jblas1765013077950375873/libquadmath-0.so: /usr/lib64/libm.so.6: version `GLIBC_2.23’ not found (required by /tmp/jblas1765013077950375873/libquadmath-0.so).
SEVERE [org.esa.snap]: JAI error occurred: ‘Problem occurs when computing a tile by the owner.’ at com.sun.media.jai.util.SunTileScheduler@4e40619b
java.lang.NullPointerException
java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dcopy(I[DII[DII)V

1 Like

@marpet, @oana_hogoiu, @MartinoF - can one of you check this. We are probably testing with Ubuntu (or similar) but it is important to also test with Redhat/CentOS/Fedora.

Martino is not anymore part of the project. Instead @FlorianD has joined.

I’m not sure if we can test with so many Unix distros. Testing would take ages to finish. But I leave this to the others.

I thought the jblas issue was solved for SNAP 8.
[SITBX-769] Investigate if we can replace jblas - JIRA (atlassian.net)
maybe @jun_lu or @lveci can check this report?

Hello,
This issue have been noticed in other projects using SNAP 8.
The GLIBC error may come from the fact that some binaries are compiled on newer Linux versions, and they may not work on older distros, like CentOS7, Ubuntu 14.04, 16.04.
On Ubuntu 20.04, after installing libgfortran5, things are OK.
So, on your CentOS 7, you can use a docker containg Ubuntu 20.04 and SNAP 8, as a workaround (but this depends how you want to use SNAP, it you need only GPT or not).

please also install libgfortran5, this has reportedly solved the issue here: Please test the SBAS_snap2stamps

Hi,

I have the same issue. Centos 7.x, SNAP 8.0.x with this error at the early beginning of the gpt graph (is this happening in reading the input?):

INFO: org.hsqldb.persist.Logger: dataFileCache open start
– org.jblas ERROR Couldn’t load copied link file: java.lang.UnsatisfiedLinkError: /tmp/jblas5430762825947091188/libquadmath-0.so: /lib64/libm.so.6: version `GLIBC_2.23’ not found (required by /tmp/jblas5430762825947091188/libquadmath-0.so).
org.jblas.NativeBlas.dgemm(CCIIID[DII[DIID[DII)V
done.
Error: [NodeId: Back-Geocoding] org.jblas.NativeBlas.dgemm(CCIIID[DII[DIID[DII)V
– org.jblas INFO Deleting /tmp/jblas5430762825947091188/libquadmath-0.so
– org.jblas INFO Deleting /tmp/jblas5430762825947091188

Your solution acctually worked for me:

Solution: at least for Linux with GLIBC older than 2.23, remove snap/s1tbx/modules/ext/org.jlinda.jlinda-* and replace with versions from SNAP 7.0.x.
i.e.: /bin/ln -fs snap_7.0.x/s1tbx/modules/ext/org.jlinda.jlinda-* snap_8.0.x/s1tbx/modules/ext/
I believe the problem is that 8.0.x jblas.jar includes a static build of libquadmath, that references a dynamic libm.so.6 at version GLIBC_2.23, which I and other distros do not have.

I was anyway wondering: is this workaround ok from a “scientific point of view”, or using SNAP 7.x modules in a SNAP 8.x processing could led to inconsistent/incongruent/invalid outputs? What are those jblas libraries for?

Thank you in advance to anyone can give me feedback!

I believe this issue has been fixed by the latest updates for SNAP/S1TBX 8.0.3 jlinda-nest and jlinda-core, nbm files dated 2021-03-12, (that include a new jblas.jar?).

Hi @s0sh0rt,

I checked both via GUI and command line and all the org.esa.s1tbx.* modules are listed at v8.0.3 and Enabled. In which location should I check the nbm files date you are referring to?

The NBM files in the update repository are automatically downloaded during update: http://step.esa.int/updatecenter/8.0/snap-toolboxes/
I may be mistaken, but thought these files had been the source of the problem. Mine are dated 3/19 which is the date I last updated SNAP.
snap/s1tbx/modules/ext/org.jlinda.jlinda-core/org-jblas/jblas.jar
snap/s1tbx/modules/ext/org.jlinda.jlinda-nest/org-jblas/jblas.jar
snap/s1tbx/update/backup/netbeans/modules/ext/org.jlinda.jlinda-nest/org-jblas/jblas.jar
snap/s1tbx/update/backup/netbeans/modules/ext/org.jlinda.jlinda-core/org-jblas/jblas.jar

Thank you Chris.

I downloaded the files but unfortunately they are the same one already present in the s1tbx module directories (indeed I installed the SNAP v8.0 yesterday on the centos 7.x machine).

I agree with you that those files are the source of the problem because in applying the workaround you suggested, my processing will succeed.

I am wondering what changes among the SNAP 7.0 and SNAP 8.0 jblas jars.

You are correct that the issue still exists. My recent test was flawed. I ran ESD from gpt command:
gpt Enhanced-Spectral-Diversity -Ssource=IW2_bg.dim
which does not error. My first post was from running ESD from the SNAP GUI, which still errors after recent updates.
I have no idea what the implications of running SNAP 8 with the older jlinda modules.
I recommend attempting command line gpt without the older jlinda modules to process your graphs.

I ran into this error again running BackGeocoding from the v8.0.x GUI without 7.0 jlinda modules (my original post). This time my solution was to simply save a graph of the same operation: product set reader - backgeocoding - write. I then ran without any errors from gpt command line. Since this error only appears when using the GUI, I believe it is safe to run with 7.0 jlinda modules, but it is probably better to run commands with gpt which does not have the issue.

1 Like

Ciao Salvatore, all,

I am having the same issue running GPT on a Centos 7 VM using SNAP 8.0.3 - however the workaround suggested by @jun_lu

produces this issue - EDIT the issue happens only when using SLC data, as it is linked to complex data:

java.lang.NoClassDefFoundError: org/jblas/ComplexDouble
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at org.esa.snap.core.gpf.internal.OperatorContext.implementsMethod(OperatorContext.java:509)
at org.esa.snap.core.gpf.internal.OperatorContext.isComputeTileMethodImplemented(OperatorContext.java:481)
at org.esa.snap.core.gpf.internal.OperatorContext.(OperatorContext.java:145)
at org.esa.snap.core.gpf.Operator.(Operator.java:96)
at org.esa.s1tbx.insar.gpf.CoherenceOp.(CoherenceOp.java:69)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.esa.snap.core.gpf.OperatorSpi.createOperator(OperatorSpi.java:134)
at org.esa.snap.core.gpf.graph.NodeContext.initOperator(NodeContext.java:132)
at org.esa.snap.core.gpf.graph.NodeContext.(NodeContext.java:51)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:80)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:58)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.jblas.ComplexDouble
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
… 42 more

any ideas about what is missing?
Thank you,
Andrea

It is possible that ComplexDouble was not present in older jblas. The GLIBC version problem is not unique to SNAP. If you can’t update you linux distro, you could try building jblas on your system. If you are using RHEL/CentoOS, then a devtoolset may be helpful if jblas expects newer tools/libraries than the standard RHEL 7 versions.

Thank you @gnwiii, however the Coherence calculation was working fine with SNAP 7 on the same machine - problems started when I had to update SNAP to v 8.0.3 to solve the change in the orbit file repo

so I used the workaround explained in this post - which seemed to work on other Centos 7 machines

I solved this issue using the workaround and using full path of the SNAP 7 s1tbx jblas jar - i.e. /bin/ln -fs /opt/snap_7.0.x/s1tbx/modules/ext/org.jlinda.jlinda-* /opt/snap_8.0.x/s1tbx/modules/ext/

@marpet @FlorianD @jun_lu @lveci
This issue is not resolved for SNAP 8.0.9/S1TBX 8.0.6 running on CentOS 7.
The jblas.jar contains dynamic links to GLIBC that is not available on CentOS 7.
This results in a number of execution errors from snap and gpt (excerpts below):

org.jblas.NativeBlas.dcopy(I[DII[DII)V
java.lang.NullPointerException
java.lang.UnsatisfiedLinkError
org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp$2.process

This file:
${snap-home}/s1tbx/modules/ext/org.jlinda.jlinda-core/org-jblas/jblas.jar
contains the two libraries below that are dynamically linked to newer GLIBC. Are these supposed to be statically linked?

ldd lib/static/Linux/amd64/libgfortran-4.so
lib/static/Linux/amd64/libgfortran-4.so: /usr/lib64/libm.so.6: version `GLIBC_2.27’ not found
(required by lib/static/Linux/amd64/libgfortran-4.so)

ldd lib/static/Linux/amd64/libquadmath-0.so
lib/static/Linux/amd64/libquadmath-0.so: /usr/lib64/libm.so.6: version `GLIBC_2.23’ not found
(required by lib/static/Linux/amd64/libquadmath-0.so)

ls -l /usr/lib64/libm.so.6
lrwxrwxrwx 1 root root 12 Dec 22 15:05 /usr/lib64/libm.so.6 → libm-2.17.so
strings /usr/lib64/libm-2.17.so | grep GLIBC_2.
GLIBC_2.2.5
GLIBC_2.4
GLIBC_2.15
GLIBC_2.15

My workaround linking the v7 jblas.jar, suggested in the first post in this thread, only worked until an update was applied, which unfortunately overwrote the linked files in v7. So linking was not the best solution. I had to download v7 again to get the jblas.jar.

If you have this issue, I recommend keeping a copy of the v7 jblas.jar so you can overwrite the file if it is required after it is replaced by an update. The workaround uses the v7 jblas.jar which is linked to older GLIBC library. Just copy the v7 file:

snap_7.0.4/s1tbx/modules/ext/org.jlinda.jlinda-core/org-jblas/jblas.jar

over both:

snap_8.0.9/s1tbx/modules/ext/org.jlinda.jlinda-core/org-jblas/jblas.jar
snap_8.0.9/s1tbx/modules/ext/org.jlinda.jlinda-nest/org-jblas/jblas.jar

Compatibility with some operating systems is still an issue with SNAP/S1TBX 9.0.1, specifically CentOS 7. The 1.2.5 version of jblas.jar contains dynamic library links which cannot be resolved on CentOS 7 since libm.so/GLIBC is older. I suggest using the 1.2.4 version of jblas.jar since this worked and was distributed with S1TBX 7.0.x, and does not contain links to loadable libraries not found on some systems.

You can either copy the S1TBX 7 version: jblas-1.2.4 (as noted in a previous post) or obtain version 1.2.4 from the Maven repo: Central Repository: org/jblas/jblas