Fail to read data with the internal GDAL

bishun945 · July 27, 2022, 9:30pm

Hi Macro,

I ran into a bug pointing to the internal GDAL (v3.2.1) when using the gpt command on my mac.

Specifically, the questionable part was found in the last chunk of “2_3_OLCI_SNAP_batch_processing_C2RCC”. I changed stderr to True in the line 34 to print the error: process = subprocess.Popen(c2rcc_processing_call, shell=True, stdout=subprocess.PIPE, stderr=True)

Error info:

-------------- Processing: --------------
/Users/Bi/learn-olci/2_OLCI_advanced/products/S3A_OL_1_EFR____20220625T083644_20220625T083944_20220626T124836_0179_087_007_1980_MAR_O_NT_002.SEN3/xfdumanifest.xml
– To: –
S3A_OL_1_EFR____20220625T083644_20220625T083944_20220626T124836_0179_087_007_1980_MAR_O_NT_002_SUBSET_IDEPIX_C2RCC.nc
– Generating config: –
/Users/Bi/learn-olci/2_OLCI_advanced/outputs/run_config.xml
– Config ready; running: –
/Applications/snap/bin/gpt /Users/Bi/learn-olci/2_OLCI_advanced/outputs/run_config.xml

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.4.2 found on system. Internal GDAL 3.2.1 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.2.1 set to be used by SNAP.
Native library load failed.
java.lang.UnsatisfiedLinkError: /Users/Bi/.snap/auxdata/gdal/gdal-3-2-1/lib/jni/libgdalalljni.dylib: dlopen(/Users/Bi/.snap/auxdata/gdal/gdal-3-2-1/lib/jni/libgdalalljni.dylib, 0x0001): Library not loaded: /usr/local/opt/jpeg/lib/libjpeg.8.dylib
Referenced from: /Users/Bi/.snap/auxdata/gdal/gdal-3-2-1/lib/libtiff.5.dylib
Reason: tried: ‘/usr/local/opt/jpeg/lib/libjpeg.8.dylib’ (no such file), ‘/usr/lib/libjpeg.8.dylib’ (no such file)
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.2.1 set to be used by SNAP.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.s3tbx.c2rcc.olci.C2rccOlciOperator: c2rcc initial tile : null, configured tile: java.awt.Dimension[width=1217,height=1023]
----- Finished running this product -----

The used xml file is uploaed: run_config.xml (2.9 KB)

You can easily get the scene name: S3A_OL_1_EFR____20220625T083644_20220625T083944_20220626T124836_0179_087_007_1980_MAR_O_NT_002

The JRE error report is also attached:
hs_err_pid43925.log (63.3 KB)

Platform: macOS Monterey v12.4
SNAP version: 9.0.0
An incompatible local GDAL was found (v3.4.2)

The error info also shows some libjpeg isseus but I checked that I already installed the jpeg dependency…

Please tell me if you need any other information from my side. And thank you in advance.

Best regards,
Shun

gnwiii · July 28, 2022, 12:14am

There are many ways to install JPEG libraries. They may differ in the file locations, compilers (clang vs gcc) and versions, supported macOS versions, use of graphics hardware, etc. It might be helpful to know how GDAL was installed, but it may be simplest to install and and test a full GDAL 3.2 package from homebrew or macports.

marpet · July 28, 2022, 7:39am

Indeed, there is a problem with GDAL and loading the libjpeg library, but this doesn’t break your processing. However, your processing does not require GDAL at all. The actual issue seems to be in the TensorFlow library which is shown in the error log. it tries to load a native lib and then crashes.
The library seems to be available but not as expected. If the library file would not be available, it would not crash but show an error message.

Do you have a Mac with M1 processor? Do you run SNAP in a virtual environment?
The log indicates this.

OS:Bsduname:Darwin 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:37 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T6000 x86_64

Maybe there are still issue when virtualising x86 application on M1 chips.
Also, the issue tracker of TensorFlow indicates that TensorFlow is not working on M1 chips.

The GDAL error might be caused by the same reason.

bishun945 · August 1, 2022, 10:02am

Thanks George for the suggestions.

Hi Macro,
thanks for the reply and sorry for my late response.

Do you have a Mac with M1 processor?

Yes, I have a Mac with M1 chip.

Do you run SNAP in a virtual environment?

No, I ran SNAP directly from Launchpad.

After some tests via GUI, the issue is probably from Idepix.Olci Operator (version 3.1.0) reporting TensorFlow could not be initialisze as you mentioned…

Anyway, thanks for your help!

Shun