Native library load failed (GDAL) when gpt called from a python subprocess

jamesrunnalls · June 27, 2023, 7:43am

Hi!

I have a strange error occurring while trying to use idepix for Sentinel 2 images, I am calling the attached XML using gpt which completes successfully when running directly from the command line however fails with the following error when called using subprocess in python.

./snap/bin/gpt idepix_MSI.xml -c 6G -e -SsourceFile=S2A_MSIL1C_20220521T101601_N0400_R065_T32TMS_20220521T154108.SAFE -PoutputFile=reproj_idepix_subset_S2A_MSIL1C_20220521T101601_N0400_R065_T32TMS_20220521T154108.SAFE.nc

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32631
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
Native library load failed.
java.lang.UnsatisfiedLinkError: /home/runnalja/.snap/auxdata/gdal/gdal-3-2-X-jni/libgdalalljni.so: libgdal.so.28: cannot open shared object file: No such file or directory
SEVERE: org.esa.snap.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
java.lang.UnsatisfiedLinkError: org.gdal.gdalconst.gdalconstJNI.GDT_Unknown_get()I
at org.gdal.gdalconst.gdalconstJNI.GDT_Unknown_get(Native Method)
at org.gdal.gdalconst.gdalconstConstants.(gdalconstConstants.java:12)
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
at java.lang.reflect.Field.get(Field.java:393)
at org.esa.snap.dataio.gdal.drivers.GDALReflection.fetchGDALLibraryConstant(GDALReflection.java:31)
at org.esa.snap.dataio.gdal.drivers.GDALConst.gaReadonly(GDALConst.java:28)
at org.esa.s2tbx.dataio.s2.ortho.Sentinel2OrthoProductReader.addBands(Sentinel2OrthoProductReader.java:592)
at org.esa.s2tbx.dataio.s2.ortho.Sentinel2OrthoProductReader.readProduct(Sentinel2OrthoProductReader.java:268)
at org.esa.s2tbx.dataio.s2.Sentinel2ProductReader.readProductNodesImpl(Sentinel2ProductReader.java:198)
at org.esa.snap.core.dataio.AbstractProductReader.readProductNodes(AbstractProductReader.java:178)
at org.esa.s2tbx.dataio.s2.ortho.Sentinel2OrthoProductReaderProxy.readProductNodes(Sentinel2OrthoProductReaderProxy.java:90)
at org.esa.snap.core.gpf.common.ReadOp.initialize(ReadOp.java:170)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:85)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:58)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:127)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:59)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
at install4j.org.esa.snap.runtime.Launcher1159904018.main(Unknown Source)

Error: org.gdal.gdalconst.gdalconstJNI.GDT_Unknown_get()I

Thanks for any insights!

James
idepix_MSI.xml (1.9 KB)

BenL · June 28, 2023, 8:59am

I am commenting on this thread as I think I am having a very similar error related to opening Sentinel-2 L2 granules via the GUI (and choosing MTD_MSIL2A.xml file):

java.lang.UnsatisfiedLinkError: org.gdal.gdalconst.gdalconstJNI.GDT_Unknown_get()I
at org.gdal.gdalconst.gdalconstJNI.GDT_Unknown_get(Native Method)
at org.gdal.gdalconst.gdalconstConstants.(gdalconstConstants.java:12)
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
at java.lang.reflect.Field.get(Field.java:393)
at org.esa.snap.dataio.gdal.drivers.GDALReflection.fetchGDALLibraryConstant(GDALReflection.java:31)
at org.esa.snap.dataio.gdal.drivers.GDALConst.gaReadonly(GDALConst.java:28)
at org.esa.s2tbx.dataio.s2.Sentinel2ProductReader.computeJP2MatrixCellsResolutionCountWithGDAL(Sentinel2ProductReader.java:295)
at org.esa.s2tbx.dataio.s2.Sentinel2ProductReader.computeJP2MatrixCellsResolutionCount(Sentinel2ProductReader.java:321)
at org.esa.s2tbx.dataio.s2.Sentinel2ProductReader.computeMatrixCellsResolutionCount(Sentinel2ProductReader.java:286)
at org.esa.s2tbx.dataio.s2.ortho.Sentinel2OrthoProductReader.addBands(Sentinel2OrthoProductReader.java:551)
at org.esa.s2tbx.dataio.s2.ortho.Sentinel2OrthoProductReader.readProduct(Sentinel2OrthoProductReader.java:265)
at org.esa.s2tbx.dataio.s2.Sentinel2ProductReader.readProductNodesImpl(Sentinel2ProductReader.java:199)
at org.esa.snap.core.dataio.AbstractProductReader.readProductNodes(AbstractProductReader.java:178)
at org.esa.s2tbx.dataio.s2.ortho.Sentinel2OrthoProductReaderProxy.readProductNodes(Sentinel2OrthoProductReaderProxy.java:90)
at org.esa.snap.core.dataio.ProductIO.readProduct(ProductIO.java:182)
at org.esa.snap.rcp.actions.file.ReadProductOperation.run(ReadProductOperation.java:61)
at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
[catch] at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)

I am running a fully up to date version of SNAP 9.0.0 on Mac OSx 11.3 (Big Sur), deliberately not upgraded due to SNAP issues on later versions.

@jamesrunnalls - i wonder if we have the same OS?

Marco_EOM · June 28, 2023, 9:12am

Hello,

I think this problem has been discussed already in this forum.
The solution starts at this comment: Cant open Sentinel 2 in SNAP 9.0.0 - s2tbx - STEP Forum (esa.int).

BenL · June 28, 2023, 9:24am

Worked like a charm for me, but I can’t speak for @jamesrunnalls. Thanks for the incredibly speedy reply, and sorry I didn’t do more digging to find the existing post/issue you linked to.

jamesrunnalls · November 7, 2023, 3:34pm

Hi Marco,

I am unsure how this solution could be applied to my case as the gpt graph works when called from the command line. It is only when using subprocess.popen that the error occurs. I also have many Sentinel 2 images that work fine using the same configuration.

Somehow calling gpt via Python is leading to a different environment than from the command line, only for certain images.

Any ideas on how this could be possible?

Cheers

James

Marco_EOM · November 8, 2023, 8:07am

It’s strange that it works for some images but for others not.
The only idea I have is to set the current working directory (cwd). With the call to popen. Set it to the bin folder of the SNAP installation.
Maybe the search for native libs finds other libraries when started from a different cwd. But just a guess.

Maybe @adraghici can have a detailed look at this.

adraghici · November 8, 2023, 9:08am

Hi @jamesrunnalls,

To understand your problem and provide a best solution we need more information from you.

Please let us:

the name and version of your Operating System
the version of Python you use
the graph file you use to process the products or the name of operators and configuration used to process the products
the name of products used to process
the LOG file of SNAP or a screenshot/console output dump file (add > ~/log_08112023.txt after your console command used to process the products using Python and send us the ~/log_08112023.txt file), after you reproduce the problem

Regards
Adrian

P.S. Thank you @Marco_EOM for tagging me on this post.

jamesrunnalls · November 20, 2023, 12:29pm

Hi Adrian,

In putting together the necessary files for you I managed to solve the problem. For some reason when GDAL version 3.2.2 is installed in the python environment where the subprocess is called from, it leads to the above error. I have upgraded GDAL (at least to 3.3.3) and the problem has disappeared.

No idea why GDAL is messing with the subprocess but at least its an easy solution.

Thanks all for your responses.

Cheers

James