Could not initialise SRTM land-water mask

animecomico · December 5, 2020, 11:33am

IdePix Classification Runtime Error:

I have a problem when execute a GPF process with IdePix with Sentinel-2 L1C data

Load the L1C Sentinel-2 Data
Subset Data
Resample Data
Call function snap_cloud_pix

Here my function:

def snap_cloud_pix(product_snap,
export=False, dir_export=‘.’, prefix_export=‘Cloud_’,
format_export=‘BEAM-DIMAP’, name_export=None):

parameters = HashMap()
parameters.put('copyToaReflectances', 'false')
parameters.put('copyFeatureValues', 'false')
parameters.put('computeMountainShadow', 'false')
parameters.put('computeCloudShadow', 'true')
parameters.put('computeCloudBuffer', 'true')
parameters.put('computeCloudBufferForCloudAmbiguous', 'true')
parameters.put('cloudBufferWidth', '2')
product_res = GPF.createProduct('Idepix.S2', parameters, product_snap)
if export:
    if name_export is None:
        name_export = 'Cloud_'
    export_product(product_res, namefile=name_export,
                   dir_export=dir_export,
                   prefix_export=prefix_export,
                   format_export=format_export)
return product_res

When execute, the problem its in GPF.createProduct and get the next error:
RuntimeError: org.esa.snap.core.gpf.OperatorException: Could not initialise SRTM land-water mask

I use SNAP v7

I don’t have any idea because is the problem.

marpet · December 5, 2020, 12:13pm

It could be that the necessary watermask data couldn’t be downloaded.
Have you tried to run Idepix in the desktop or from the command line with gpt?

animecomico · December 5, 2020, 12:49pm

Hi, I run in command line GPT using Python, and I have the same error in desktop.

This is the error:
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.zip.ZipFile.(ZipFile.java:169)
at org.esa.snap.watermask.operator.SRTMOpImage.(SRTMOpImage.java:69)
at org.esa.snap.watermask.operator.SRTMOpImage.create(SRTMOpImage.java:56)
at org.esa.snap.watermask.operator.WatermaskClassifier.createCenterImage(WatermaskClassifier.java:196)
at org.esa.snap.watermask.operator.WatermaskClassifier.(WatermaskClassifier.java:99)
at org.esa.snap.idepix.s2msi.S2IdepixClassificationOp.initialize(S2IdepixClassificationOp.java:212)
Caused: org.esa.snap.core.gpf.OperatorException: Could not initialise SRTM land-water mask
at org.esa.snap.idepix.s2msi.S2IdepixClassificationOp.initialize(S2IdepixClassificationOp.java:216)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:507)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:294)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.GPF.createProductNS(GPF.java:330)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:305)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:284)
at org.esa.snap.idepix.s2msi.S2IdepixOp.processSentinel2(S2IdepixOp.java:154)
at org.esa.snap.idepix.s2msi.S2IdepixOp.initialize(S2IdepixOp.java:143)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:507)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:294)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.GPF.createProductNS(GPF.java:330)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:305)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:284)
at org.esa.snap.core.gpf.ui.DefaultSingleTargetProductDialog.createTargetProduct(DefaultSingleTargetProductDialog.java:138)
[catch] at org.esa.snap.core.gpf.ui.SingleTargetProductDialog.onApply(SingleTargetProductDialog.java:131)
at org.esa.snap.ui.AbstractDialog.lambda$initUI$6(AbstractDialog.java:519)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
at java.awt.Component.processMouseEvent(Component.java:6539)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6304)
at java.awt.Container.processEvent(Container.java:2239)
at java.awt.Component.dispatchEventImpl(Component.java:4889)
at java.awt.Container.dispatchEventImpl(Container.java:2297)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4535)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4476)
at java.awt.Container.dispatchEventImpl(Container.java:2283)
at java.awt.Window.dispatchEventImpl(Window.java:2746)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
at java.awt.EventQueue$4.run(EventQueue.java:733)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:159)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

marpet · December 5, 2020, 12:55pm

In your user directory is the data located. For me it is:
C:\Users\Marco.snap\auxdata\watermask.

You can try if you can unzip those files manually. I guess something went wrong during the download and the files are corrupted.

bart-lg · December 5, 2020, 12:59pm

This is related to my post

Seems that the automatic download of SRTM ZIP-files is currently failing…

animecomico · December 5, 2020, 1:02pm

Hi, yes, the data is corrupted, the size of .zip is different of yours:

So, How resolve this? Deleting the files .zip?

animecomico · December 5, 2020, 1:03pm

Yes, the same problem

bart-lg · December 5, 2020, 1:09pm

If you know the exact URL where you can download those zip files try to remove the corrupted zip-files and to download them manually and to place them in the same folder. Now it should work.

Seems that for some reason the SRTM zip downloads seem not to work out if triggered by SNAP directly.

Since no updates have been run in my case since it worked, I guess that this is a server problem.

marpet · December 5, 2020, 1:10pm

Yes, deleting the files and running the processing again should solve the problem. The downloads are started again.

There was the restart of the STEP website. Probably this caused the issues.

bart-lg · December 5, 2020, 1:13pm

In my case deleting the zip-files and run it again did not work out. Same problem as before. I’m running currently a script with multiple locations and I always get the same error.

Only manual download helped so far. And that is what I am doing the whole day. Cancelling the script if the error occured, downloading the SRTM zip file manually and start GPT again.

animecomico · December 5, 2020, 1:19pm

I dont know the URL of files

animecomico · December 5, 2020, 1:20pm

Ok, I tried deleting the files

bart-lg · December 5, 2020, 1:26pm

In my case the URL was included in the error message as you can see. In your JAVA error message this is not the case. So maybe there is another log-file where the error is specified more precisely including the URL.

You can also try to create a XML file and use GPT to run your process. Then you should get an error message like mine and there you’ll find the appropriate URL.

In this document you will find how to create the XML file and how to use GPT.

animecomico · December 5, 2020, 1:41pm

Hi, Nice, I tried to use the XML file, thanks

animecomico · December 5, 2020, 5:37pm

Hi, I delete the files, but the issue continue, the files are corrupted

animecomico · December 15, 2020, 10:44am

Hi, I solve my problem, this my code in Python

import os
import requests
import zipfile

def check_watermask_files(home_path='/home/carsf/',
                          dir_temp='/home/carsf/snapcheckfiles/',
                          dir_watermask='.snap/auxdata/watermask/',
                          url_download='https://step.esa.int/auxdata/watermask/images/'):
    files2test = ['50m.zip', '150m.zip',
                  'GC_water_mask.zip',
                  'MODIS_north_water_mask.zip',
                  'MODIS_south_water_mask.zip']
    if not os.path.isdir(dir_temp):
        try:
            os.mkdir(dir_temp)
        except OSError:
            print("Creation of the directory %s failed" % dir_temp)
        else:
            print("Successfully created the directory %s " % dir_temp)
    for fileintest in files2test:
        locate_test = home_path + '/' + dir_watermask + '/' + fileintest
        try:
            the_zip_file = zipfile.ZipFile(locate_test)
            ret = the_zip_file.testzip()
        except Exception as error:
            print('Watermask Error {}'.format(error))
            ret = -1
        if ret is not None:
            print("First bad file in zip: {}".format(ret))
            path_url_file = url_download + fileintest
            print("Try download data {}".format(path_url_file))
            is_down, size_file = is_downloadable(path_url_file)
            print('The file is Downloadable: {}'.format(is_down))
            print('The file size is {}'.format(size_file))
            if is_down:
                print('Downloading {}'.format(fileintest))
                r = requests.get(path_url_file, allow_redirects=True)
                open(locate_test, 'wb').write(r.content)
                print('Downloaded {} in {}'.format(fileintest, locate_test))
            else:
                print('Cant Download file data')
                return False
        else:
            print("Zip file is good {}".format(locate_test))
    return True


def is_downloadable(url):
    """
    Does the url contain a downloadable resource
    """
    h = requests.head(url, allow_redirects=True)
    header = h.headers
    content_type = header.get('content-type')
    if 'text' in content_type.lower():
        return False, None
    if 'html' in content_type.lower():
        return False, None
    content_length = header.get('content-length', None)
    return True, content_length

This working for me.