Docker image loop error on ElevationFile

alattaruolo · March 1, 2024, 9:58am

Hello,
I hope this is the right place for this question. I am working with a docker image in which I install the Sentinel Toolboxes version 9.0.0 from the link https://download.esa.int/step/snap/9.0/installers/esa-snap_sentinel_unix_9_0_0.sh.
Here is my DockerFile

FROM python:3.9
RUN apt-get update 
RUN apt-get install libgfortran5
#download snap installer version 9.0
RUN wget https://download.esa.int/step/snap/9.0/installers/esa-snap_sentinel_unix_9_0_0.sh
#change file execution rights for snap installer
RUN chmod +x esa-snap_sentinel_unix_9_0_0.sh
#install snap with gpt
RUN ./esa-snap_sentinel_unix_9_0_0.sh -q
#link gpt so it can be used systemwide
RUN ln -s /usr/local/snap/bin/gpt /usr/bin/gpt

RUN snap --nosplash --nogui --modules --update-all 2>&1 | while read -r line; do echo "$line"; [ "$line" = "updates=0" ] && sleep 2 && pkill -TERM -f "snap/jre/bin/java"; done; exit 0

These are the steps that I had found to make at least run the image,

the first library libgfortran5 to install was suggested in this link- >How to fix "ImportError: libgfortran.so.5: cannot open shared object file: No such file or directory." in ubuntu 18.04 aarch64 - Stack Overflow (after a few attempts I found this library helped)
then for installation I followed the following guide on the forum → Update snap in docker container, which he had problems for updating the snap version which is solved by the snap --nosplash --nogui --modules --update-all ...

The main problem that I have is this in this image I try to execute the following GPT command (the products that I am using and the .xml are correctly seen by the image it’s not a problem of these):

gpt assets/s1coherence.xml -Pimage1_fpath=files/S1A_IW_SLC__1SDV_20231124T171520_20231124T171547_051362_0632A5_BE31.SAFE -Pimage2_fpath=files/S1A_IW_SLC__1SDV_20231206T171520_20231206T171547_051537_0638A9_99C8.SAFE -Psubswath=IW3 -Pburst1=4 -Pburst2=4 -Pcoherence_fpath=files/preprocess/demo_work.tif

But the gpt standard output is the following:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/S1A/2023/11/S1A_OPER_AUX_POEORB_OPOD_20231214T071058_V20231123T225942_20231125T005942.EOF.zip
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/S1A/2023/12/S1A_OPER_AUX_POEORB_OPOD_20231226T071109_V20231205T225942_20231207T005942.EOF.zip
...10%....INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip

Tt looks like the execution is starting but then it goes in a loop trying to retrieve a org.esa.snap.core.dataop.dem.ElevationFile, which is very similar to an issue on Infinity loop while downloading SRTM files with VV and VH layer but my concern here is that I am using SNAP v 9 while on the post it was version 8. I do have this problem just inside a Docker image, on my local computer after installing snap I was able to execute the command, I guess there should be a configuration to add or fix in the Dockerfile but now sure what.

LudvikAdamec · March 8, 2024, 1:46pm

We are facing the same issues with our Docker setup.

The SNAP version inside Docker has been updated to the latest available.

Is there anything we can do to prevent these issues?

Additionally, is there a plan to release a fix or a new version of ESA SNAP? We are experiencing the same issues on all machines, including Linux and Windows host environments.

Log from SNAP

INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_43_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_43_03.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_43_03.zip

Our Docker image, similar to the one described above, worked successfully for several months, processing hundreds of images without any issues.

alfredoahds · March 12, 2024, 1:16pm

We’re seeing the same issue with our previously working docker setup - is there any update on this problem?

alattaruolo · March 12, 2024, 3:16pm

My guess is that it will be fixed with the new release of the toolbox (but this is just my supposition). In the meantime which Docker configurations are you guys using? I found these workarounds but not sure if they are the best solutions.

RUN apt-get install libgfortran5

and also my update ends with a exit 0; since every time it gives me an error

RUN snap --nosplash --nogui --modules --update-all 2>&1 | while read -r line; do echo "$line"; [ "$line" = "updates=0" ] && sleep 2 && pkill -TERM -f "snap/jre/bin/java"; done; exit 0

diana_harosa · March 13, 2024, 3:34pm

@djagula can you have a look ? Thanks

LudvikAdamec · March 15, 2024, 8:58am

We have tested running the GPT with an XML graph locally from the command line on Windows without Docker. The test resulted in the same infinite loop when retrieving DEM as described above.

It appears that this issue is not only limited to Docker but also affects SRTM 3Sec data. Our current solution was to switch from SRTM 3Sec to SRTM 1Sec HGT. The 1Sec data seems to be working without issues both inside and outside of Docker.

martina_m · March 27, 2024, 4:05pm

Hello, we are seeing the same issue downloading elevation files using SNAP v9 in linux within docker:

INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_47_17.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_47_17.zip on https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_47_17.zip
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_47_17.zip
WARNING: org.esa.snap.core.dataop.dem.ElevationFile: http error:https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_47_17.zip on https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_47_17.zip

It appears that this specific file doesn’t exist, however we had this happen with plenty of files that do exist. Downloading them and saving them in .snap/auxdata/dem/‘SRTM 3 Sec’ folder helps, however this one we have no workaround for as it doesn’t seem to exist. Is there a new url where these files are located? Is there any update on this issue?

diana_harosa · March 29, 2024, 9:36am

Hello @martina_m,
SRTM90 has been updated with the missing files. Could you please confirm if they are accessible now?

For all others existing files, have you tried to test the link from outside docker container? Are you able to download them?

alattaruolo · March 29, 2024, 10:28am

Hello @diana_harosa I tried to execute the coherence as I wrote in the beginning but I still have the same problem, always inside the docker container which I recreated from scartch , I saw that from the browser the SRMT90 (https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_39_03.zip) is downloaded but inside the container it loops.

alattaruolo · April 2, 2024, 8:47am

Hello with the help of @diana_harosa I was able to make things work inside the docker container

To be able to understand if this is a SNAP issue or a server-side issue.
In the <SNAP_INSTALL_FOLDER>/etc/snap.auxdata.properties file, could you replace the following line:
DEM.srtm3GeoTiffDEM_HTTP = https://download.esa.int/step/auxdata/dem/SRTM90/tiff/
with
DEM.srtm3GeoTiffDEM_HTTP = https://step.esa.int/auxdata/dem/SRTM90/tiff/ ?

This is what has been suggested as possible solution (which worked for my issue) and inside the docker container I added the following line after the SNAP update (as final command):

RUN sed -i 's/https:\/\/download.esa.int\/step\/auxdata\/dem\/SRTM90\/tiff\//https:\/\/step.esa.int\/auxdata\/dem\/SRTM90\/tiff\//g' /usr/local/snap/etc/snap.auxdata.properties

diana_harosa · April 2, 2024, 2:03pm

JIRA ticket SNAP-3669 created to track this issue.

martina_m · April 22, 2024, 11:12am

Hi @diana_harosa, thank you for the update, the downloads of the elevation files seem to be behaving better and not getting stuck, but we are still having problems with SNAP trying to look for file srtm_47_17.zip which doesn’t exist. It keeps trying to download it (same as the example I posted in previous post). Any idea why it tries to download this file as it’s not in the SRTM90 list at all?

To answer your questions, we are able to download all the files outside of the docker container directly from the link, but not the srtm_47_17.zip file as it does not exist.