SRTM ZIP-files are corrupted or not found

bart-lg · December 5, 2020, 12:56pm

Hi everybody!

I am using the Graph Processing Tool (GPT) of SNAP on Linux to preprocess and subset Sentinel-1 images. This worked well until yesterday midnight. Suddenly there is a problem with new downloaded DEM SRTM auxdata. If the files exist from a processing job in the past for the given subset, everything works fine. If new files are required, SNAP downloads them automatically, but the downloaded ZIP-files are suddenly corrupted and GPT stays in an endless loop trying to unpack the new ZIP-file:

INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N40W080.SRTMGL1.hgt.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: error in opening zip file
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N40W080.SRTMGL1.hgt.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: error in opening zip file
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N40W080.SRTMGL1.hgt.zip
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: error in opening zip file
[...]

After killing the GPT process and testing the ZIP-file, I can see that it is corrupted:

~/.snap/auxdata/dem/SRTM 1Sec HGT$ unzip -t N40W080.SRTMGL1.hgt.zip
Archive:  N40W080.SRTMGL1.hgt.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of N40W080.SRTMGL1.hgt.zip or
        N40W080.SRTMGL1.hgt.zip.zip, and cannot find N40W080.SRTMGL1.hgt.zip.ZIP, period.

If I remove the corrupted ZIP-file and download it manually using wget, the ZIP-file is totally fine:

~/.snap/auxdata/dem/SRTM 1Sec HGT$ rm N40W080.SRTMGL1.hgt.zip
~/.snap/auxdata/dem/SRTM 1Sec HGT$ wget http://step.esa.int/auxdata/dem/SRTMGL1/N40W080.SRTMGL1.hgt.zip
--2020-12-05 13:05:27--  http://step.esa.int/auxdata/dem/SRTMGL1/N40W080.SRTMGL1.hgt.zip
Resolving step.esa.int (step.esa.int)... 37.252.127.80
Connecting to step.esa.int (step.esa.int)|37.252.127.80|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://step.esa.int/auxdata/dem/SRTMGL1/N40W080.SRTMGL1.hgt.zip [following]
--2020-12-05 13:05:27--  https://step.esa.int/auxdata/dem/SRTMGL1/N40W080.SRTMGL1.hgt.zip
Connecting to step.esa.int (step.esa.int)|37.252.127.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9572548 (9.1M) [application/zip]
Saving to: ‘N40W080.SRTMGL1.hgt.zip’

N40W080.SRTMGL1.hgt.zip                 100%[=============================================================================>]   9.13M  6.50MB/s    in 1.4s

2020-12-05 13:05:28 (6.50 MB/s) - ‘N40W080.SRTMGL1.hgt.zip’ saved [9572548/9572548]

~/.snap/auxdata/dem/SRTM 1Sec HGT$ unzip -t N40W080.SRTMGL1.hgt.zip
Archive:  N40W080.SRTMGL1.hgt.zip
    testing: N40W080.hgt              OK
No errors detected in compressed data of N40W080.SRTMGL1.hgt.zip.

If I start the GPT process again, no error occurs and the preprocessing and creation of the subset works fine.

I tried this on two machines:
Ubuntu 18.04 with SNAP 7
Ubuntu 20.04 with SNAP 8

The problem occurs on both systems since last night.

Does someone encounter a similar problem or knows what the problem could be?

Thanks!

AriJeannin · December 7, 2020, 9:56am

Yes, same problem here!

suhendra0812 · December 6, 2020, 12:39pm

Hi, I am trying to execute the Range Doppler Terrain Correction on Sentinel-1 data using SRTM 3Sec DEM data. But, the download process of the DEM data is repeated continuously and the error appears such as the following picture. I just encounter this issue recently, I have no issue like this before. The message log is attached as well.

[messages.log|attachment]

The message log:
(upload://53U0MujG7ZUkQrkq2w9MWBupHpk.log) (76.0 KB)

ABraun · December 6, 2020, 12:41pm

have you tried SRTM 1Sec instead?

suhendra0812 · December 6, 2020, 1:10pm

@ABraun Yes, I did. It also raised an error, but with a different circumstance. It raised “error in opening zip file”. When I checked to auxdata directory, the downloaded DEM data just had 1 kB filesize.

ABraun · December 6, 2020, 1:13pm

Is it possible that your disk is full?

suhendra0812 · December 6, 2020, 1:20pm

I don’t think so.

I have tried to reinstall the software, but the error still remains.

FYI, this is my specification:
Manufacturer: Dell Inc.
Model: Inspiron 5575
Processor: AMD Ryzen 5 2500U with Radeon Vega Mobile Gfx 2.00 GHz
Installed RAM: 8,00 GB (7,64 GB usable)
System type: 64-bit operating system, x64-based processor
OS: Windows 10 Home (10.0.19042 Build 19042)

ABraun · December 6, 2020, 1:24pm

maybe your firewall blocks the correct download.
Please make sure that port 443 is open in your firewall settings. How to Open a Port on Windows Firewall
Windows and Mac: How to open ports in your firewall

suhendra0812 · December 6, 2020, 1:55pm

Still raised the error

ABraun · December 6, 2020, 2:03pm

would it work to manually download the DEM and use it as an external DEM for the terrain correction?

suhendra0812 · December 6, 2020, 2:09pm

Correct me if I put it in the wrong way.

bart-lg · December 6, 2020, 2:12pm

Got the same problem with downloading of SRTM files in SNAP.

This topic is related to

and

There has been a server restart on Saturday evening. Since then there are problems with fetching SRTM files in SNAP. Manual downloads outside SNAP are working. Seems for me that this is a server issue.

ABraun · December 6, 2020, 2:55pm

please exctract the data before you use it as an external DEM. SNAP needs it as GeoTiff in WGS84

karavias · December 6, 2020, 3:05pm

Hello there! I have the same problem on back-geocoding. I tried to use an external DEM as GeoTiff and I still get an error message.

ismail · December 7, 2020, 6:27am

same problem…???

vincentschut · December 7, 2020, 8:17am

I think snap’s ability to open zip files is broken since yesterday (or so). We get the same error on the “ww15mgh_b.zip” file (the data for the EGM96 gravitational model):

2020-12-05T19:15:15.651809346Z INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip
2020-12-05T19:15:15.651816240Z error in opening zip file                            
2020-12-05T19:15:15.651823153Z 90% done.                                                                                               
2020-12-05T19:15:15.651829433Z org.esa.snap.core.gpf.OperatorException: error in opening zip file

Downloading and unzipping that same file locally works perfectly fine, though…

ABraun · December 7, 2020, 9:32am

@lveci could you please have a look?

Santos · December 6, 2020, 8:23am

Dear s1tbx team,

I get this error when computing interferogram using SRTM 1Sec HGT:
SEVERE: org.esa.snap.core.dataop.dem.ElevationFile: error in opening zip file
INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving http://step.esa.int/auxdata/dem/SRTMGL1/N37W010.SRTMGL1.hgt.zip

I tried to access the link http://step.esa.int/auxdata/dem/SRTMGL1/, and find the file N37W010.SRTMGL1.hgt.zip, but I didn’t find.

Thank you in advance for your attention

Santos · December 6, 2020, 1:26pm

I’m processing interferograms with SLC products from sentinel 1 A, referring ≈ 1 year (7/26/2019 - 6/26/2020).
I’m using SRTM 1Sec HGT in Back-Geocoding and TopoPhaseRemoval.

ABraun · December 7, 2020, 4:36pm

7 posts were split to a new topic: SNAP searches non-existent SRTM tiles