Using mask file with KMeansClusterAnalysis in python

johnb · June 9, 2016, 6:50pm

How do I assign a mask from a shapefile when using KMeansClusterAnalysis in python:

I have the following so far, but I get the error below:

source = ProductIO.readProduct(mask_path)
parameters = HashMap()

parameters.put('vectorFile', river_mask_file)
target = GPF.createProduct('Import-Vector', parameters, source)

source = ProductIO.readProduct(in_file)
parameters.put('clusterCount', 30)
parameters.put('iterationCount', 30)
parameters.put('randomSeed', 314150)
parameters.put('roiMaskName', 'MASK_River_dissolve')
parameters.put('sourceBandNames', in_band)
target = GPF.createProduct('KMeansClusterAnalysis', parameters, source)
ProductIO.writeProduct(target, out_file, 'GeoTIFF')

Error:
target = GPF.createProduct(‘Import-Vector’, parameters, source)
RuntimeError: ambiguous Java method call, too many matching method overloads found

It works in the GUI version by dragging the shapefile in, but I’d like it to work in python.
Any help would be much appreciated.
Thanks
John

marpet · June 14, 2016, 7:17am

Where does mask_path point to? Maybe it is not a product which can be read by SNAP.
Is source properly initialised after doing this call? I think it could be None.

johnb · June 14, 2016, 7:18pm

If the ‘source’ is initialised with the geotif file that I want masked… see below, I get the following error:
target = GPF.createProduct(‘Import-Vector’, parameters, source)
RuntimeError: org.esa.snap.core.gpf.OperatorException: com/bc/ceres/swing/figure
/FigureStyle

I have tried initialised ‘source’ with the shapefile but I get the following error:
target = GPF.createProduct(‘Import-Vector’, parameters, source)
RuntimeError: ambiguous Java method call, too many matching method overloads found

parameters = HashMap()
in_file = r"D:\John\projects\river\input.tif"
river_mask_file = r"D:\John\projects\river\input.tif"
source = ProductIO.readProduct(in_file)
parameters.put('vectorFile', river_mask_file)
target = GPF.createProduct('Import-Vector', parameters, source)
source = ProductIO.readProduct(in_file)
parameters.put('clusterCount', 30)
parameters.put('iterationCount', 30)
parameters.put('randomSeed', 314150)
parameters.put('roiMaskName', 'MASK_SlaveRiver_dissolve')
parameters.put('sourceBandNames', in_band)
target = GPF.createProduct('KMeansClusterAnalysis', parameters, source)
ProductIO.writeProduct(target, out_file, 'GeoTIFF')

marpet · June 15, 2016, 6:07am

river_mask_file needs to be a shapefile (*.shp).
The in_file should be OK as it is.
For the call of KMeansClusterAnalysis you should exchange source by target. sourceis the original unchanged file.

johnb · June 15, 2016, 3:21pm

Hi Marc
Sorry I had a typo in my question … the river mask was defined as a shapefile:
river_mask_file = r"D:\John\projects\river\input.shp"
and
in_file = r"D:\John\projects\river\input.tif"
source = ProductIO.readProduct(in_file)

and I get the following error:

target = GPF.createProduct(‘Import-Vector’, parameters, source)
RuntimeError: org.esa.snap.core.gpf.OperatorException: com/bc/ceres/swing/figure
/FigureStyle

marpet · June 16, 2016, 6:44am

I think this is caused by an error in the snappy initialisation.

Please have a look in the __init__.py file of snappy.

You will see the following code at line 25-26

EXCLUDED_DIR_NAMES = {'org.esa.snap.snap-worldwind', 'org.esa.snap.snap-rcp', 'org.esa.snap.snap-product-library',
                  'org.esa.snap.ceres-ui', 'org.esa.snap.snap-sta-ui'}

Please remove the ‘org.esa.snap.ceres-ui’ from the list.

Afterwards it should work

johnb · June 16, 2016, 1:21pm

Hi Marco
I removed ‘org.esa.snap.ceres-ui’ from line 25-26 of init.py, but I’m getting the same error:
target = GPF.createProduct(‘Import-Vector’, parameters, source)
RuntimeError: org.esa.snap.core.gpf.OperatorException: com/bc/ceres/swing/figure
/FigureStyle
Any other suggestions?
Thanks
John

marpet · June 16, 2016, 2:53pm

Probably this wasn’t enough, the needed jar file is still not loaded.
In the same file try to change following lines (90-93)

...
if os.path.isfile(path) and name.endswith('.jar'):
if not (name.endswith('-ui.jar') or name in EXCLUDED_JAR_NAMES):
    env[0].append(path)
elif os.path.isdir(path) and name not in EXCLUDED_DIR_NAMES:
...

to

...
if os.path.isfile(path) and name.endswith('.jar'):
    env[0].append(path)
elif os.path.isdir(path):
...

johnb · June 16, 2016, 7:13pm

Thanks Marco
After making the changes that you suggested to init.py the following python worked successfully :

parameters = HashMap()
source = ProductIO.readProduct(in_file)
parameters.put('vectorFile', river_mask_file)
parameters.put('separateShapes', False)
target1 = GPF.createProduct('Import-Vector', parameters, source)

parameters.put('clusterCount', 30)
parameters.put('iterationCount', 30)
parameters.put('randomSeed', 314150)
parameters.put('roiMaskName', 'MASK_SlaveRiver_dissolve')
parameters.put('sourceBandNames', in_band)
target2 = GPF.createProduct('KMeansClusterAnalysis', parameters, target1)
ProductIO.writeProduct(target2, out_file, 'GeoTIFF')

I did get the following warning after making your suggested changes to init.py (any suggestions to get rid of these warnings?)

WARNING: org.esa.snap.SnapCoreActivator: org.esa.snap.core.gpf.OperatorSpi: Prov
ider org.esa.snap.python.gpf.PyOperatorSpi could not be instantiated
java.lang.NoClassDefFoundError: org/openide/util/Lookup
at org.esa.snap.rcp.util.NbResourceLocator.getResourceClassLoader(NbReso
urceLocator.java:22)
at com.bc.ceres.core.DefaultResourceLocator.locateResources(DefaultResou
rceLocator.java:31)
at com.bc.ceres.core.ResourceLocator.getResources(ResourceLocator.java:3
4)
at org.esa.snap.core.util.ServiceFinder.scanClassPath(ServiceFinder.java
:195)
at org.esa.snap.core.util.ServiceFinder.findServices(ServiceFinder.java:
148)
at org.esa.snap.python.gpf.PyOperatorSpi.(PyOperatorSpi.java:42)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou

rce)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at java.util.ServiceLoader$LazyIterator.nextService(Unknown Source)
at java.util.ServiceLoader$LazyIterator.next(Unknown Source)
at java.util.ServiceLoader$1.next(Unknown Source)
at org.esa.snap.SnapCoreActivator.loadServices(SnapCoreActivator.java:60
)
at org.esa.snap.core.gpf.internal.OperatorSpiRegistryImpl.loadOperatorSp
is(OperatorSpiRegistryImpl.java:77)
Caused by: java.lang.ClassNotFoundException: org.openide.util.Lookup
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
… 16 more

Start processing…

johnb · June 16, 2016, 7:24pm

Hi Marco
Is it possible to obtain any statistical info (mean, SD, etc… for each input layer) under each class generated by KMeansClusterAnalysis?
Let me know
Thanks for all your help
John

marpet · June 17, 2016, 7:20am

OK, good that it is running. The error might occurs because we now include to many modules. So we must be more specific again.

...
if os.path.isfile(path) and name.endswith('.jar'):
if name.endswith('ceres-ui.jar') or (not (name.endswith('-ui.jar') orname in EXCLUDED_JAR_NAMES)):
    env[0].append(path)
elif os.path.isdir(path) and name not in EXCLUDED_DIR_NAMES:
...

Probably you also have to remove ‘org.esa.snap.ceres-ui’ from the EXCLUDED_DIR_NAMES again.

In the metadata you find a section named Cluster_Analysis
There you find for each class the cluster centers of each of the input bands.
But more information is not provided.

johnb · June 17, 2016, 1:22pm

Awesome, the code changes worked, no more errors

Where is the metadata located? It only creates a geotiff, if I run gdalinfo on the geotiff I get the following metadata info:

Metadata domains:
(default)
IMAGE_STRUCTURE
Metadata:
AREA_OR_POINT=Area
TIFFTAG_IMAGEDESCRIPTION=FQ17W_20151214_135725_class_snappy
TIFFTAG_RESOLUTIONUNIT=1 (unitless)
TIFFTAG_XRESOLUTION=1
TIFFTAG_YRESOLUTION=1

marpet · June 29, 2016, 7:27pm

If you open the file with SNAP you should see the metadata.
For external applications it might be a bit complicated to access this metadata.
SNAP writes some XML into a Tiff_Tag field with the number 65000 (if I remember correctly). Inside this the Cluster_Analysis metadata is stored.

kkozarev · February 13, 2019, 12:43pm

Hello,

I had the same issues as above, and followed the suggested changes to init.py:

Remove ‘org.esa.snap.ceres-ui’ from EXCLUDED_DIR_NAMES
Change
if os.path.isfile(path) and name.endswith(’.jar’):
if not (name.endswith(’-ui.jar’) or name in EXCLUDED_JAR_NAMES):
env[0][name] = path
elif os.path.isdir(path) and name not in EXCLUDED_DIR_NAMES:
to
if os.path.isfile(path) and name.endswith(’.jar’):
env[0][name] = path
elif os.path.isdir(path):

I have the latest version of SNAP installed.
When I try to import snappy functionality I get the following:

>>> from snappy import ProductIO, HashMap, GPF

WARNING: org.esa.snap.core.util.ServiceLoader: org.esa.snap.core.gpf.OperatorSpi: Provider org.esa.snap.python.gpf.PyOperatorSpi could not be instantiated

java.lang.NoClassDefFoundError: org/openide/util/Lookup

at org.esa.snap.rcp.util.NbResourceLocator.getResourceClassLoader(NbResourceLocator.java:22)

at com.bc.ceres.core.DefaultResourceLocator.locateResources(DefaultResourceLocator.java:31)

at com.bc.ceres.core.ResourceLocator.getResources(ResourceLocator.java:34)

at org.esa.snap.core.util.ServiceFinder.scanClassPath(ServiceFinder.java:195)

at org.esa.snap.core.util.ServiceFinder.findServices(ServiceFinder.java:148)

at org.esa.snap.python.gpf.PyOperatorSpi.<clinit>(PyOperatorSpi.java:44)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at java.lang.Class.newInstance(Class.java:442)

at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)

at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)

at java.util.ServiceLoader$1.next(ServiceLoader.java:480)

at org.esa.snap.core.util.ServiceLoader.loadServices(ServiceLoader.java:40)

at org.esa.snap.core.gpf.internal.OperatorSpiRegistryImpl.loadOperatorSpis(OperatorSpiRegistryImpl.java:77)

at org.esa.snap.core.gpf.GPF.<clinit>(GPF.java:97)

Caused by: java.lang.ClassNotFoundException: org.openide.util.Lookup

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

… 17 more

Traceback (most recent call last):

File “<stdin>”, line 1, in <module>

File “/Users/kamen/anaconda2/lib/python2.7/site-packages/snappy/init.py”, line 395, in <module>

Engine.start()

RuntimeError: java.util.ServiceConfigurationError: org.esa.snap.runtime.Activator: Provider org.esa.snap.worldwind.WWActivator could not be instantiated

kkozarev · February 13, 2019, 2:49pm

Also, I’d like to point out that another issue may be that the function call
sentinel_1 = ProductIO.readProduct(output_dir + ‘manifest.xml’)
returns a NoneType object…