Connection refused when downloading DEM

[jirikadlec2]

I am using SNAP GPT to process backscatter from SLC. The process is run on multiple machines in parallel (each machine is processing a different SLC scene).

Since today morning all of the processes fail with error:

INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip
SEVERE: org.esa.snap.engine_utilities.download.DownloadableContentImpl: Failed to download remote file.
java.net.ConnectException: Connection refused (Connection refused)

All of the machines are connecting to the Internet via external IP address 45.130.28.26.

When I tried to run command:

wget http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip

Then I also get a “connection refused” error.

On the other hand, if I run the same process on a different machine with another external IP 45.130.30.156, then the DEM download is successful without any error.

Is it possible that my IP 45.130.28.26 was blocked by the step.esa.int server due to too many requests? What do I need to do to unblock the IP?

[oana_hogoiu]

Hello,
Yes, your IP is currently banned.
There are some rules and limitations enforced:

Snap connection to server

Hello @dmb891 , Yes, most probably your IP was banned (you depassed the number of retries for unsuccessful requests during the configured time interval) A banned IP will be whitelisted after a specific amount of time, which is 24 hours. In addition, if you are behind a NAT (like in some companies, or even Internet Service Providers), there is also a limit for the maximum number allowed of open connections (at a time) from a single IP address.

[jirikadlec2]

@Snap connection to server

In addition, if you are behind a NAT (like in some companies, or even Internet Service Providers), there is also a limit for the maximum number allowed of open connections (at a time) from a single IP address

Hello, Thank you for the reply. I have stopped the processes and will retry after 24 hours on a smaller number of machines and also add some delay between the SNAP GPT calls to reduce the number of simultaneous open connections.
Do you have information, what is the maximum number of allowed open connections at a time from a single IP address?

[oana_hogoiu]

The maximum number of allowed connections at a time from a singe IP address is 10.

[cristianoLopes]

@jirikadlec2

have stopped the processes and will retry after 24 hours on a smaller number of machines and also add some delay between the SNAP GPT calls to reduce the number of simultaneous open connections.

If you have a large number of machines, you should actually consider running a local mirror! We are only keeping a copy of these files in STEP for convenience and last resort. People should use the official distribution channels as much as possible and people doing cluster parallel runs should consider running their own local mirror (specially because of network and security considerations that most of the times apply).

[jirikadlec2]

@cristianoLopes

If you have a large number of machines, you should actually consider running a local mirror!

Thank you for the suggestion. Is there any tutorial available, how to (1) set up a local mirror and (2) how to configure SNAP GPT to use my local mirror instead of the default STEP server?
The processes using Internet resources from STEP are Apply-Orbit-File and Terrain-Correction. Apply-Orbit-File downloads some AUX files and Terrain-Correction downloads srtm ZIP files. Any suggestion, what the structure and naming of the orbit files and srtm ZIP files is needed for the local mirror to work, would help!

[AVollrath]

Dears,

so I seem to run into a similar issue. I must have been blocked as I was downloading my SRTM s upfront (from the step server of course) to create a mirror myself. Now I am using Copernicus DEM, after having figured out that they reside on Amazon and are not limited.

However, while I solved the issue with DEMs, during co-registration of 2 bursts I get a connection refused error. If I read the error right, it checks for a new Snap version during co-registration? See attached error log.
20220204_A43_IW2_7635_coreg.err_log (3 KB)

Any suggestion how to run parallel processing on a big machine without running into this issue? Setting up a cluster is not an option for me, and neither sequential processing is, as it would simply take too long

Thanks,
Andreas