How to disable egm96 download and all auxillary data download?

I get this error when running my gpt graph

# rave at Ryans-MacBook-Pro.local in ~/ms-sar/preprocessing on git:main ✖︎ [16:43:33]
→ docker run snap       
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at java.net.Socket.connect(Socket.java:556)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at java.net.URL.openStream(URL.java:1067)
	at org.esa.snap.core.util.VersionChecker.getRemoteVersion(VersionChecker.java:144)
	at org.esa.snap.core.util.VersionChecker.checkForNewRelease(VersionChecker.java:92)
	at org.esa.snap.core.util.EngineVersionCheckActivator.start(EngineVersionCheckActivator.java:21)
	at org.esa.snap.runtime.Engine.informActivators(Engine.java:222)
	at org.esa.snap.runtime.Engine.lambda$start$0(Engine.java:121)
	at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
	at org.esa.snap.runtime.Engine.start(Engine.java:121)
	at org.esa.snap.runtime.Engine.start(Engine.java:90)
	at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
	at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
	at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
	at install4j.org.esa.snap.runtime.Launcher1159904018.main(Unknown Source)
WARNING: org.esa.snap.core.util.VersionChecker: Not able to check for new SNAP version. Remote version could not be retrieved.
Executing processing graph
INFO: org.esa.s1tbx.commons.io.ImageIOFile: Using FileCacheImageInputStream
INFO: org.hsqldb.persist.Logger: dataFileCache open start
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
Exception calling QC Rest API:  Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
WARNING: org.esa.s1tbx.sar.gpf.orbits.ApplyOrbitFileOp: ApplyOrbit ignoring error and continuing: org.apache.http.conn.HttpHostConnectException: Connect to qc.sentinel1.eo.esa.int:443 [qc.sentinel1.eo.esa.int/131.176.235.71] failed: Operation timed out (Connection timed out)
version = 2.36
INFO: java.util.prefs.FileSystemPreferences$1: Created user preferences directory.
.WARNING: org.esa.snap.dataio.netcdf.util.MetadataUtils: Missing configuration property 'snap.dataio.netcdf.metadataElementLimit'. Using default (100).
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip
SEVERE: org.esa.snap.engine_utilities.download.DownloadableContentImpl: Failed to download remote file.
java.net.ConnectException: Connection refused (Connection refused)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at org.esa.snap.engine_utilities.download.DownloadableContentImpl.downloadFile(DownloadableContentImpl.java:247)
	at org.esa.snap.engine_utilities.download.DownloadableContentImpl.lambda$getRemoteHttpFile$0(DownloadableContentImpl.java:224)
	at org.esa.snap.engine_utilities.download.DownloadableContentImpl.getRemoteHttpFile(DownloadableContentImpl.java:229)
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.<init>(EarthGravitationalModel96.java:98)
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.create(EarthGravitationalModel96.java:85)
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.instance(EarthGravitationalModel96.java:77)
	at org.esa.snap.dem.dataio.FileElevationTile.<init>(FileElevationTile.java:42)
	at org.esa.snap.dem.dataio.FileElevationModel.init(FileElevationModel.java:64)
	at org.esa.snap.dem.dataio.FileElevationModel.<init>(FileElevationModel.java:49)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.getElevationModel(RangeDopplerGeocodingOp.java:483)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.computeTileStack(RangeDopplerGeocodingOp.java:843)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
	at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
	at javax.media.jai.OpImage.getTile(OpImage.java:1129)
	at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
	at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
	at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
	at org.esa.snap.raster.gpf.LinearTodBOp.computeTile(LinearTodBOp.java:116)
	at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
	at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
	at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
	at javax.media.jai.OpImage.getTile(OpImage.java:1129)
	at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
	at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:116)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
	at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
	at javax.media.jai.OpImage.getTile(OpImage.java:1129)
	at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
	at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at java.net.Socket.connect(Socket.java:556)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3061)
	at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)
	at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)
	at java.net.URLConnection.getContentLength(URLConnection.java:485)
	at org.esa.snap.engine_utilities.download.DownloadableContentImpl.downloadFile(DownloadableContentImpl.java:246)
	... 34 more

Unable to download http://step.esa.int/auxdata/dem/egm96/ to null
java.lang.NullPointerException
90% done.
org.esa.snap.core.gpf.OperatorException: Unable to download http://step.esa.int/auxdata/dem/egm96/ to null
	at org.esa.snap.core.gpf.graph.GraphProcessor$GPFImagingListener.errorOccurred(GraphProcessor.java:379)
	at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(SunTileScheduler.java:1646)
	at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:921)
	at javax.media.jai.OpImage.getTile(OpImage.java:1129)
	at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
	at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
	at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
	at org.esa.snap.raster.gpf.LinearTodBOp.computeTile(LinearTodBOp.java:116)
	at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
	at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
	at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
	at javax.media.jai.OpImage.getTile(OpImage.java:1129)
	at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
	at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
	at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:116)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
	at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
	at javax.media.jai.OpImage.getTile(OpImage.java:1129)
	at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
	at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
Caused by: org.esa.snap.core.gpf.OperatorException: Unable to download http://step.esa.int/auxdata/dem/egm96/ to null
	at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.computeTileStack(RangeDopplerGeocodingOp.java:1067)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
	at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
	at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
	... 21 more
Caused by: org.esa.snap.core.gpf.OperatorException: Unable to download http://step.esa.int/auxdata/dem/egm96/ to null
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.computeTileStack(RangeDopplerGeocodingOp.java:846)
	... 24 more
Caused by: java.io.IOException: Unable to download http://step.esa.int/auxdata/dem/egm96/ to null
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.<init>(EarthGravitationalModel96.java:100)
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.create(EarthGravitationalModel96.java:85)
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.instance(EarthGravitationalModel96.java:77)
	at org.esa.snap.dem.dataio.FileElevationTile.<init>(FileElevationTile.java:42)
	at org.esa.snap.dem.dataio.FileElevationModel.init(FileElevationModel.java:64)
	at org.esa.snap.dem.dataio.FileElevationModel.<init>(FileElevationModel.java:49)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.getElevationModel(RangeDopplerGeocodingOp.java:483)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.computeTileStack(RangeDopplerGeocodingOp.java:843)
	... 24 more

Error: Unable to download http://step.esa.int/auxdata/dem/egm96/ to null

But I turned off the use of egm in my .xml (not sure what it is, I am using an external DEM). How do I stop this from downloading? I am trying to get all my external data, including orbit files set up ahead of time and turn off unneeded downloads so that my docker container doesn’t need to depend on servers going down.

here is the part of my xml where I turned egm off. There’s no other egm setting in my xml or gpt command

			<demName>External DEM</demName>
			<externalDEMFile>${DEM_FILE_PATH}</externalDEMFile>
			<externalDEMNoDataValue>9999.0</externalDEMNoDataValue>
			<externalDEMApplyEGM>false</externalDEMApplyEGM>

I also tried to disable aux download with gpt but it doesn’t seem to do anything:

/usr/local/snap/bin/gpt -e -Dsnap.gpf.allowAuxdataDownload=false /usr/src/S1_GRD_preprocessing_external_dem.xml -Presolution=30 -Porigin=5 -Pfilter='Refined Lee' -PDEM_FILE_PATH=$3 -Pcrs='GEOGCS["WGS84(DD)", DATUM["WGS84", SPHEROID["WGS84", 6378137.0, 298.257223563]], PRIMEM["Greenwich", 0.0], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH]]' -Pinput=$1 -Poutput=$2
 

@marpet can you please check that property… a quick check on github seems to indicate that this property is actually only used by the WatermaskUtils operator… nothing else.

Is there to turn off external connections altogether? Inc. new updates and aux data. Like Ryan indicates this is an interesting approach for very controlled environments like docker instances that try to use a local repo (which I understand may not be an easy setup anyway).

2 Likes

Yes, the property snap.gpf.allowAuxdataDownload is not yet used at all places.

In this case @lveci should check the code.

Caused by: java.io.IOException: Unable to download http://step.esa.int/auxdata/dem/egm96/ to null
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.<init>(EarthGravitationalModel96.java:100)
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.create(EarthGravitationalModel96.java:85)
	at org.esa.snap.dem.dataio.EarthGravitationalModel96.instance(EarthGravitationalModel96.java:77)
	at org.esa.snap.dem.dataio.FileElevationTile.<init>(FileElevationTile.java:42)
	at org.esa.snap.dem.dataio.FileElevationModel.init(FileElevationModel.java:64)
	at org.esa.snap.dem.dataio.FileElevationModel.<init>(FileElevationModel.java:49)
	at org.esa.s1tbx.sar.gpf.geometric.RangeDopplerGeocodingOp.getElevationModel(RangeDopplerGeocodingOp.java:483)

In FileElevationTile:42 an instance of the EarthGravitationalModel96 is created, regardless of the configuration. Maybe this is necessary?

But the download is only started if no auxdata file is found locally.
So, there must be the file <Auxdata_Path>dem/rgm96/ww15mgh_b.zip

Direct download of this file is available here: http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip

1 Like

Thank you @marpet. A Feature Request: It’d be great to have a document describing for each required external data source where it needs to be placed in the .snap folder and where it can be downloaded from.