GDAL commands broken in docker image

liendolucas · June 24, 2022, 12:32pm

Hi there!

I’m trying to deploy my app to Docker and decided to use the mundialis/esa-snap:ubuntu image. At some point during my pipeline I need to call the gdal_translate command. To test things out I first run: docker run --rm -it mundialis/esa-snap:ubuntu and then run a simply gdal_translate. It comes back with the following error:

gdal_translate: error while loading shared libraries: libgdal.so.26: cannot open shared object file: No such file or directory

It seems that some (or all) binaries under: /root/.snap/auxdata/gdal/gdal-3-0-0/bin are broken. I played around a bit making some symlinks but that didn’t fixed the problem. The libraries seems to exist though and are present at: /root/.snap/auxdata/gdal/gdal-3-0-0/lib.

Any help much appreciated.

Thanks for your time,
Lucas.

gnwiii · June 25, 2022, 2:05pm

Running complex applications as “root” is discouraged. Seeing /root/.snap/ may point to the underlying source of problems where a user process doesn’t have access to this
directory.

liendolucas · June 26, 2022, 4:57pm

I don’t quite understand what you mean. @gnwiii Yeah, it’s running as root in a Docker container, it might not be best practice, but things should just work anyways. Now, I said, ok I will generate the Docker image from scratch by myself. Installed everything (SNAP, JPY, SNAPPY) now when I run my app I get:

INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.4 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.4 set to be used by SNAP.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.4 set to be used by SNAP.
INFO: org.hsqldb.persist.Logger: dataFileCache open start

100% done.

100% done.

100% done.
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-csi-srtm.openterrain.org.s3.amazonaws.com/source/srtm_40_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_40_03.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://cgiar-csi-srtm.openterrain.org.s3.amazonaws.com/source/srtm_40_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_40_03.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: java.lang.reflect.InvocationTargetException

The above error repeating ad infinitum. At this point my frustration is beyond imagination. On one side I have the mundialis Docker image run SNAP ok but GDAL broken. On mine I have GDAL but the app won’t start due to extra resources fetching errors. I also have no detailed information on the SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: java.lang.reflect.InvocationTargetException exception which tells me nothing, so I can’t even have an idea the real source of the problem… The first URL when I try to perform a wget gives me: “HTTP request sent, awaiting response… 403 Forbidden”. Why on top of almost 1 Gb of installation I also need to download things on the fly when launching? Is there a way I can I trigger the fetch of these resources manually so I can at least have them all as part of the image?

Thanks.

gnwiii · June 27, 2022, 12:59am

See https://senbox.atlassian.net/wiki/spaces/SNAP/pages/1898053693/SNAP+FAQs#A-process-related-to-digital-elevation-models-is-taking-forever-to-finish.

Running non-system software as root is not reliable. Linux distros often use a “hardened” configuration for root’s environment, so non-system software often breaks. It is often easier to run SNAP with a normal user configuration that to work around the
distro configurations and updates may break you work-around.

liendolucas · June 27, 2022, 12:08pm

@gnwiii No luck. Changed the URL as mentioned there, now I’m getting a similar message but with the new URL:

INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.4 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.4 set to be used by SNAP.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.4 set to be used by SNAP.
INFO: org.esa.s1tbx.commons.io.ImageIOFile: Using FileCacheImageInputStream
INFO: org.hsqldb.persist.Logger: dataFileCache open start

100% done.

100% done.

100% done.
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_40_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_40_03.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: java.lang.reflect.InvocationTargetException
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_40_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://srtm.csi.cgiar.org/wp-content/uploads/files/srtm_5x5/TIFF/srtm_40_03.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: java.lang.reflect.InvocationTargetException

Again performing a manual wget from my machine, gives me back a 403:

wget "http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_40_03.zip"
--2022-06-27 14:06:25--  http://skywatch-auxdata.s3-us-west-2.amazonaws.com/dem/SRTM90/tiff/srtm_40_03.zip
Resolving skywatch-auxdata.s3-us-west-2.amazonaws.com (skywatch-auxdata.s3-us-west-2.amazonaws.com)... 52.218.213.89
Connecting to skywatch-auxdata.s3-us-west-2.amazonaws.com (skywatch-auxdata.s3-us-west-2.amazonaws.com)|52.218.213.89|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-06-27 14:06:26 ERROR 403: Forbidden.

gnwiii · June 27, 2022, 6:02pm

The SRTM 3sec data seems to move around. The FAQ suggests using RTM 1sec data.

NASA EOSDIS has NASADEM. EOSDIS has SRTM 1sec and 3sec data. You will need an EarthData login.

neteler · June 28, 2022, 1:46pm

When we developed the Dockerfile, we had to use the GDAL version bundled with SNAP:

github.com

mundialis/esa-snap/blob/ubuntu/Dockerfile#L47

      
        
            
            
# install SNAPPY
            RUN apt-get install default-jdk maven -y
            ENV JAVA_HOME "/usr/lib/jvm/java-11-openjdk-amd64/"
            RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1
            COPY snap /src/snap
            RUN bash /src/snap/install.sh
            RUN update-alternatives --remove python /usr/bin/python3
            
            
# due to Ubuntu-GDAL being too old we prefer to use the SNAP-bundled GDAL:
            # INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
            
            
# path
            RUN echo "export PATH=\$PATH:/usr/local/snap/bin/:/root/.snap/auxdata/gdal/gdal-3-0-0/bin" >> /root/.bashrc
            
            
# tests
            # https://senbox.atlassian.net/wiki/spaces/SNAP/pages/50855941/Configure+Python+to+use+the+SNAP-Python+snappy+interface
            RUN (cd /root/.snap/snap-python/snappy && python3 setup.py install)
            RUN /usr/bin/python3 -c 'from snappy import ProductIO'
            RUN /usr/bin/python3 /src/snap/about.py
            RUN /root/.snap/auxdata/gdal/gdal-3-0-0/bin/gdal-config --version

Please suggest changes as needed, ideally as a pull request in the GitHub repository.

oana_hogoiu · June 28, 2022, 2:13pm

Hello,
If I understood correctly, you are trying to use in the command line the SNAP internal GDAL. Therefore you are trying to call outside SNAP something that is internal to SNAP …
Running SNAP internal GDAL from a process external to SNAP is not guaranted to work.

You could make it work, but there are some changes needed in your environment variables:

PATH should contain the path towards GDAL binaries within SNAP
GDAL_DATA should contain the path towards the share/gdal directory within the internal SNAP GDAL distribution
GDAL_PLUGINS should contain the path towards gdalplugins directory within the internal SNAP GDAL distribution
PROJ_LIB should contain the path towards share/proj directory within the internal SNAP GDAL distribution
GDAL_PAM_ENABLED=NO

liendolucas · June 28, 2022, 2:26pm

Ok, so basically that GDAL is “internal” and if I understand correctly it’s part of SNAP and is NOT intended to be used in isolation, nor to be called from elsewhere? So what I should be trying out is just to simply run: apt-get install gdal-bin in the image and check if the global os gdal binaries supplied by Ubuntu just work…

oana_hogoiu · June 28, 2022, 2:31pm

Yes, try to install GDAL (latest version) on your OS.

liendolucas · June 28, 2022, 2:40pm

Ok, just installing the gdal-bin package leaves the binaries in place. I’ve tested calling gdal_translate and it works. Thanks everyone for the time and patience answering the thread!