Missing Python Libraries?

snap Sen-ET
#1

I’m trying to run the steps to calculate ET according to the user manual. I have installed the snappy python package. When I try to run the Prepare ERA5 Reanalysis operator to the download ERA5 data, I get the following python error:

Traceback (most recent call last):
  File "/home/kavok/.snap/auxdata/sen-et-conda-Linux64/sen-et-snap-scripts/ecmwf_data_preparation.py", line 6, in <module>
    import snappy_utils as su
  File "/home/kavok/.snap/auxdata/sen-et-conda-Linux64/sen-et-snap-scripts/snappy_utils.py", line 12, in <module>
    from snappy import ProductIO, Product, ProductData, ProductUtils, String
  File "/home/kavok/.snap/snap-python/snappy/__init__.py", line 65, in <module>
    import jpy
ImportError: dynamic module does not define module export function (PyInit_jpy)
Process exited with value 1
Finished tool execution in 2 seconds

I attach a screenshot:

image
image1224×837 34 KB

If you would to check the ERA5 input I’m using, it can be downloaded here:
https://drive.google.com/drive/folders/1e0Hdpfi9m1QDgXkmKz_HWRcj3Y8OreJn?usp=sharing

I have tried contacting the developers of this plugin (dhigroup.com) but they have not responded. It’s pretty disappointing that support seems to be so lacking for this plugin. Anybody know if there is an office at ESA that can step up?

#2

There are at least 3 Python packages that answer to the name snappy, so it is all too easy to end up with conflicting packages. It can be helpful to mention your OS and Python version, see: How to report bugs effectively.

Did you install SNAP snappy using <snap_install_dir>/bin/snappy-conf? Can you run the provided test and example scripts (note that some tests require downloading a data file)?

Try searching the forum for previous reports such as this link.

#3

Hi, thanks for your response.

My system is a linux Ubuntu 20.04 LTS.
I did install snappy with snappy-conf with python2.7:

/opt/snap/bin/snappy-conf /usr/bin/python2.7

I just ran the tests that come in the tests/ directory in the snappy installation. They all completed succesfully. For example:

kavok@kavok-laptop:/tmp$ python2.7 ~/.snap/snap-python/snappy/tests/test_snappy_perf.py 

SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder '.'.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
('Band.readPixels(): w =', 1121L, ', dtype=np.float32:', 305L, 'calls in', 435.3599548339844, 'ms, that is ', 1.4321051145854748, 'ms per call')
.('Band.readValidMask(): w =', 1121L, ', dtype=np.bool:', 305L, 'calls in', 97.9471206665039, 'ms, that is ', 0.32219447587665756, 'ms per call')
.
----------------------------------------------------------------------
Ran 2 tests in 1.204s

OK

How can I check that SNAP is actually calling the same python2.7 that I used to install snappy?

In any case, this seems to be an issue with the code of the Sen-ET plugin itself (if I see the error message). I was hoping the plugin developers would be available for support…

Thanks.

#4

The Sen-ET manual (p. 17) says “Additionally, SNAP should be configured for use with Python 3.6.”. Also:

Blockquote
3.4.4 Import error - snappy / jpy
If during the execution of the Sen-ET operators, the user experiences an import error for
either snappy or jpy, it means that SNAP was not configured properly for use with SNAP.
In order to fix this issue, the user will have to re-install SNAP and configure SNAP for
use with Python. The Python version has to be 3.6 in order to match with the Conda
environment that we distribute.

You might be able to find the conda environment in ~/.snap and use that, or you can install Anaconda python (currently 3.8) and then add the python 3.6 environment. This has the advantage that you can use python 3 in any snappy scripts you write (python 2 is going away).

#5

Hi, I did try to install it with python 3.6 but this happens:

Configuring SNAP-Python interface...
java.io.IOException: Python configuration failed.
Command [/usr/bin/python3.6 ./snappyutil.py --snap_home /opt/snap --java_module /opt/snap/snap/modules/org-esa-snap-snap-python.jar --force --log_file ./snappyutil.log --jvm_max_mem 10G --java_home /opt/snap/jre --req_arch amd64]
failed with return code 30.
Please check the log file '/home/kavok/.snap/snap-python/snappy/snappyutil.log'.
	at org.esa.snap.python.PyBridge.configureJpy(PyBridge.java:232)
	at org.esa.snap.python.PyBridge.installPythonModule(PyBridge.java:149)
	at org.esa.snap.rcp.cli.SnapArgsProcessor.processPython(SnapArgsProcessor.java:103)
	at org.esa.snap.rcp.cli.SnapArgsProcessor.process(SnapArgsProcessor.java:49)
	at org.netbeans.modules.sendopts.DefaultProcessor.process(DefaultProcessor.java:202)
	at org.netbeans.spi.sendopts.Option$1.process(Option.java:387)
	at org.netbeans.api.sendopts.CommandLine.process(CommandLine.java:317)
	at org.netbeans.modules.sendopts.HandlerImpl.execute(HandlerImpl.java:62)
	at org.netbeans.modules.sendopts.Handler.cli(Handler.java:69)
	at org.netbeans.CLIHandler.notifyHandlers(CLIHandler.java:234)
	at org.netbeans.core.startup.CLICoreBridge.cli(CLICoreBridge.java:82)
	at org.netbeans.CLIHandler.notifyHandlers(CLIHandler.java:234)
	at org.netbeans.CLIHandler$1.exec(CLIHandler.java:268)
	at org.netbeans.CLIHandler.finishInitialization(CLIHandler.java:447)
	at org.netbeans.MainImpl.finishInitialization(MainImpl.java:256)
	at org.netbeans.Main.finishInitialization(Main.java:92)
	at org.netbeans.core.startup.Main.start(Main.java:316)
	at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:123)
	at java.lang.Thread.run(Thread.java:748)
Python configuration error: Python configuration failed.
Command [/usr/bin/python3.6 ./snappyutil.py --snap_home /opt/snap --java_module /opt/snap/snap/modules/org-esa-snap-snap-python.jar --force --log_file ./snappyutil.log --jvm_max_mem 10G --java_home /opt/snap/jre --req_arch amd64]
failed with return code 30.
Please check the log file '/home/kavok/.snap/snap-python/snappy/snappyutil.log'.

Unfortunately the error does not really tell much on what went wrong. The contents of the snappyutil.log are the following:

INFO: Installing from Java module '/opt/snap/snap/modules/org-esa-snap-snap-python.jar'
WARNING: Architecture requirement possibly not met: Python is x86_64 but JVM requires amd64
INFO: Installing jpy...
INFO: Extracting 'lib/jpy-0.9.0-cp36-cp36m-linux_x86_64.whl' from '/opt/snap/snap/modules/org-esa-snap-snap-python.jar'
INFO: Unzipping '/home/kavok/.snap/snap-python/snappy/lib/jpy-0.9.0-cp36-cp36m-linux_x86_64.whl'
INFO: Configuring jpy...
INFO: jpy Python API configuration written to '/home/kavok/.snap/snap-python/snappy/jpyconfig.py'
INFO: jpy Java API configuration written to '/home/kavok/.snap/snap-python/snappy/jpyconfig.properties'
INFO: Configuring snappy...
INFO: snappy configuration written to '/home/kavok/.snap/snap-python/snappy/snappy.ini'
INFO: Importing snappy for final test...
ERROR: Configuration failed with exit code 30

The contents of the snappy.ini:

[DEFAULT]
snap_home = /opt/snap
java_max_mem: 10G
# snap_start_engine: False
# java_class_path: ./target/classes
# java_library_path: ./lib
# java_options: -Djava.awt.headless=false
# debug: False

I do see that there are files in the ~/.snap/snap-python directory. However, trying to run the tests give an error.

kavok@kavok-laptop:~$ python3.6  ~/.snap/snap-python/snappy/tests/test_snappy_mem.py 
Traceback (most recent call last):
  File "/home/kavok/.snap/snap-python/snappy/tests/test_snappy_mem.py", line 5, in <module>
    import snappy
ModuleNotFoundError: No module named 'snappy'

So I’m at a loss here, on whether this is a python error, SNAP error, snappy error or… Seems that it does not complete the installation of the library in the python environment.

#6

It would be good to mention how you installed python 3.6 in Ubuntu 20.04. Also, it looks like you have snap in /opt/snap, which often means you installed using “root” privileges (e.g., using sudo). This can cause odd problems due to permissions. If you did this because you have more than one user there can be conflicts when on user installs updates while another user isn’t expecting a change.

After you configure snappy you have to arrange for python to find it. This is the “Configure Python” section of Configure Python to use the SNAP-Python snappy interface.

The USERPROFILE method works on linux if you replace USERPROFILE with HOME (or, in your system, /opt/snap). I prefer this as it makes it clear where python expects to find ESA snappy, so won’t break if one of the other snappy modules gets installed.

#7

Hi, I installed python3.6 using apt-get:

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get install python3.6

And the the configuration of snappy failed as described above.

/opt/snap/bin/snappy-conf /usr/bin/python3.6

However it does seem to write the python library into the snappy directory, just failed to set it up in the python local repository. So I went to the directory and tried to install it myself:

cd ~/.snap/snap-python/snappy
sudo python3.6 setup.py install

No errors, but when I try to import snappy…

Python 3.6.11 (default, Jun 29 2020, 04:25:06) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import snappy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/snappy/__init__.py", line 65, in <module>
    import jpy
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

So it seems it also needs the python development libraries.

sudo apt-get install libpython3.6-dev

I proceed to run it again, and now there are no errors when importing.

Python 3.6.11 (default, Jun 29 2020, 04:25:06) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import snappy
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder '.'.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.

And all the tests complete succesfully.

python3.6  ~/.snap/snap-python/snappy/tests/test_snappy_mem.py
python3.6  ~/.snap/snap-python/snappy/tests/test_snappy_product.py
python3.6  ~/.snap/snap-python/snappy/tests/test_snappy_perf.py

I think the error messages and output from SNAP could be made much clearer. The java exception shown gives no clue as what has gone wrong.

Thanks for the help.

#8

Glad you got it going. The issues you encountered stem from use of a 3rd party python on an OS that is much newer than your SNAP, so you can understand why they are difficult for ESA’s developers to anticipate.

There are drawbacks to installing snappy to the system library, including conflicts if one of the other packages called “snappy” is installed or if other users have different SNAP versions installed, and is an extra step when installing a new SNAP version. Setting the PYTHONPATH is better, but I prefer to have each python script use sys.path.append(snappy_dir). This makes it clear that the script uses SNAP’s snappy so isn’t affected if a different snappy gets installed (e,g., as a dependency of some other python package).

1 Like