Problem installing Python Snappy on a Mac

development python
#1

Hello,

I’d very much appreciate your help. I’ve spent several hours trying to install the Python Snappy library on my Mac, but have run into issues. I can see people have had similar problems in the past, but I’m not sure if they managed to solve theirs.

Here’s my set up:
Macbook Pro Retina 15" from 2015
macOS BigSur Version 11.2.3

I did the following steps.

source ~/.bash_profile
conda activate
conda env create -f /Users/olivierlejeune/dev/Coding/Sentinel/environment.yml

In the last setup page, I tried to configure my python installation to use Snappy by checking the tickbox and proving the path to my “sentinel” virtual environment, but ran into the following error:

error=13, Permission denied

Running the same commands in Terminal hasn’t worked either.

conda activate sentinel
cd /Applications/snap/bin
./snappy-conf /opt/anaconda3/envs/sentinel

I get the following error:
Configuring SNAP-Python interface…
java.io.IOException: Cannot run program “/opt/anaconda3/envs/sentinel” (in directory “/opt/anaconda3/envs/sentinel/lib/snappy”): error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.esa.snap.python.PyBridge.configureJpy(PyBridge.java:228)
at org.esa.snap.python.PyBridge.installPythonModule(PyBridge.java:149)
at org.esa.snap.rcp.cli.SnapArgsProcessor.processPython(SnapArgsProcessor.java:103)
at org.esa.snap.rcp.cli.SnapArgsProcessor.process(SnapArgsProcessor.java:49)
at org.netbeans.modules.sendopts.DefaultProcessor.process(DefaultProcessor.java:202)
at org.netbeans.spi.sendopts.Option$1.process(Option.java:387)
at org.netbeans.api.sendopts.CommandLine.process(CommandLine.java:317)
at org.netbeans.modules.sendopts.HandlerImpl.execute(HandlerImpl.java:62)
at org.netbeans.modules.sendopts.Handler.cli(Handler.java:69)
at org.netbeans.CLIHandler.notifyHandlers(CLIHandler.java:234)
at org.netbeans.core.startup.CLICoreBridge.cli(CLICoreBridge.java:82)
at org.netbeans.CLIHandler.notifyHandlers(CLIHandler.java:234)
at org.netbeans.CLIHandler$1.exec(CLIHandler.java:268)
at org.netbeans.CLIHandler.finishInitialization(CLIHandler.java:447)
at org.netbeans.MainImpl.finishInitialization(MainImpl.java:256)
at org.netbeans.Main.finishInitialization(Main.java:92)
at org.netbeans.core.startup.Main.start(Main.java:316)
at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:123)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
… 19 more
Python configuration error: Cannot run program “/opt/anaconda3/envs/sentinel” (in directory “/opt/anaconda3/envs/sentinel/lib/snappy”): error=13, Permission denied

Any help you could provide would be hugely appreciated.

Thanks a bunch.

Olivier

#2

I’m retired from a lab where macs were widely used, but don’t currently have access to macOS. Apple has made changes to increase security of recent macOS versions so there could be some new issues, but I am familiar with some of the complexities of configuring ESA SNAP snappy on macOS.

@ OlivierLej I did the following steps.

You have done a good job of describing what you have done, but there are a few details that need clarification. Installing a JDK should be a last resort if you can’t get snappy to run with Anacoda3’s Python-3.6 environment. OpenJDK comes in two forms: the Java Runtime Environment (JRE) and the full Development Kit (OpenJDK JDK) which could be needed to build
a jpy for a Python version later than 3.6.

  • In Terminal, set JAVE_HOME variable to the following
    export JAVA_HOME=$(/usr/libexec/java_home)

Typo (JAVE --> JAVA)? Again, you should not do this until you have failed to get ESA SNAP snappy working with Python-3.6.

  • Created an Anaconda virtual environment called sentinel with Python version 3.6.

source ~/.bash_profile
conda activate
conda env create -f /Users/olivierlejeune/dev/Coding/Sentinel/environment.yml

Some users have tried this but ended up with a very old Python-3.6. There appear to be newer versions on conda-forge. In a terminal, running just
python should give a banner and the python > prompt. This banner should mention the version (3.6.x) and Anaconda.

Did run run the installer using sudo? Some users use sudo to install SNAP in /Applications, others install to ~/Applications which does not require use of sudo. If you installed using sudo you may encounter permission problems when configuring snappy or updating SNAP.

In the last setup page, I tried to configure my python installation to use Snappy by checking the tickbox and proving the path to my “sentinel” virtual environment, but ran into the following error:

error=13, Permission denied
If you used the “path” given below, that won’t work.

Running the same commands in Terminal hasn’t worked either.

conda activate sentinel
cd /Applications/snap/bin
./snappy-conf /opt/anaconda3/envs/sentinel

The “path” should end with python, e.g., /opt/anaconda3/envs/sentinel/python. Check that this path is working in the terminal.

I get the following error:
Configuring SNAP-Python interface…
java.io.IOException: Cannot run program “/opt/anaconda3/envs/sentinel” (in directory “/opt/anaconda3/envs/sentinel/lib/snappy”): error=13, Permission denied

This is because snappy-conf tried to execute the snappy directory as if it was a python program. It might be useful to have snappy-conf check that the python-path ends with a python program and display the startup header. I find users are often surprised when they don’t get the expected Python version.

Potential troublespots to consider:

  • Apple will not distribute a current bash version due to the license restrictions. Recent versions of macOS do provide zsh, which many consider an improvement over bash, but which may object to some bash scripts (the fixes are usually trivial).

  • There are many Python libraries called “snappy”, so care is needed to ensure that python loads the ESA SNAP version from ~/.snap/snap-python. It is safest to set the python search path to include this directory in each script that needs SNAP snappy. Attempting to install ESA SNAP snappy into a python distribution may fail in another snappy is present, or may break other python scripts that expect a different snappy.

  • Other users have some success building jpy for current Python version, but your use case may be different. With most SNAP snappy users working with the provided jpy binary wheels, those have been well-tested and it is easy for others to reproduce any bugs you might encounter.

Please let us know how you make out. If the documentation is unclear or
misleading for macOS users please explain the issue.

#3

Hi George,

Many thanks for your help, you’ve managed to make me understand where I was going wrong.

I managed to install the Snap software tool (version 8.0.0) and link it with my Anaconda environment, by using your suggested path:
/opt/anaconda3/envs/sentinel/bin/python, instead of the wrong one I was using previously.

As advised in the Snap wiki, I was then able to make snappy permanently accessible, by installing it into my Python installation:

source ~/.bash_profile
conda activate sentinel
cd /Users/olivierlejeune/.snap/snap-python/snappy
python setup.py install

This gave me the following:

running install
running build
running build_py
creating build
creating build/lib
creating build/lib/snappy
copying snappy/init.py -> build/lib/snappy
copying snappy/setup.py -> build/lib/snappy
copying snappy/snappyutil.py -> build/lib/snappy
running install_lib
creating /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy
copying build/lib/snappy/init.py -> /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy
copying build/lib/snappy/setup.py -> /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy
copying build/lib/snappy/snappyutil.py -> /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy
byte-compiling /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy/init.py to init.cpython-36.pyc
byte-compiling /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy/setup.py to setup.cpython-36.pyc
byte-compiling /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy/snappyutil.py to snappyutil.cpython-36.pyc
running install_egg_info
Writing /opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy-8.0.0-py3.6.egg-info

However, just as I thought I’d finally managed to crack my installation problem, it threw a new error.

I entered:

source ~/.bash_profile
conda activate sentinel
python
import snappy

It gave the following error :disappointed_relieved:

Traceback (most recent call last):
** File “”, line 1, in **
** File “/opt/anaconda3/envs/sentinel/lib/python3.6/site-packages/snappy/init.py”, line 62, in **
** import jpyutil**
ModuleNotFoundError: No module named 'jpyutil’

I have tried some fo the workarounds mentioned in the wiki, but no luck. It seems that, once again, I’m not the only user facing this error, but I remain unsure how to solve it in my case. Should I now try to install jpy on its own?

Thanks again for your time, you’ve been extremely helpful.

Olivier

#4

Can you show us the banner that appears when you run python-3.6 in a
terminal? This may be useful for comparing with other users’ installations.

After the successful snappy_util.py install, were you able to run the test as described in the wiki article?

On Windows, I get:

C:\Users\XXXXX\.snap\snap-python>python
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import snappy
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
>>>

macOS should be similar, and the build date may help pin down problems with jpy.

I don’t recommend this approach because of the potential conflicts with one of the other packages called “snappy”. You can’t predict when some
other python package could try to use ESA SNAP snappy when attempting to use a different snappy package. The final option given in the wiki, using the Windows USERPROFILE environment variable, also works well for macOS and linux with HOME replacing USERPROFILE. This makes it
easier to share scripts with other users (assuming they kept the default install paths).

For other macOS users, the tests described in the wiki article fail because the jpy wheel provided by ESA SNAP was created with a different Python
binary (probably using a newer macOS version than the one used to create their anaconda3 python-3.6). There are many Python-3.6 wheels on conda-forge. I haven’t discovered a way to find out which macOS version was used to build them, but most appear to be several years old.

If you have macports, homebrew, or fink, you may be able to install a Python-3.6 binary that uses your system’s compiler runtime libraries. As Apple tries to support older software, this may allow you to use ESA SNAP’s jpy wheel.

#5

Hi again George,

When I run:
source ~/.bash_profile
conda activate sentinel
python

I get:
Python 3.6.13 | packaged by conda-forge | (default, Feb 19 2021, 05:36:16)
[GCC Clang 11.0.1] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.

#6

That is a recent build, so there is good chance it is compatible with the bundled jpy. Did the simple tests work?

#7

Hello,

I’ve uninstalled and re-installed everything, and tried to configure my snappy installation manually, and uncovered the following error, which seems related:

INFO: Installing from Java module '/Applications/snap/snap/modules/org-esa-snap-snap-python.jar’
INFO: Installing jpy…
ERROR: The module ‘jpy’ is required to run snappy, but no binary ‘jpy’ wheel matching the pattern
’jpy-{version}-cp36-{abi_tag}-macosx_10_9_x86_64.whl’ could be found.
You can try to build a ‘jpy’ wheel yourself, then copy it into
"/Users/olivierlejeune/.snap/snap-python/snappy", and then run the configuration again.
Unzip the jpy sources in /Users/olivierlejeune/.snap/snap-python/snappy/jpy-.zip, then
** cd jpy-<version>** ** python setup.py bdist_wheel**
** cp dist/*.whl "/Users/olivierlejeune/.snap/snap-python/snappy"** **Or get the source code from https://github.com/bcdev/jpy and follow the build instructions:** ** git clone https://github.com/bcdev/jpy.git**
** $ cd jpy**
ERROR: Configuration failed with exit code 10

#8

The macsox-..._x86_64.whl is your Python’s search pattern for a “platform wheel” (“platform” because it contains compiled code that will only work on specific platforms).

The -cp36- shows you were successful in using a Python-3.6, but
the macosx_10_9 indicates your Python-3.6 uses macOS 10.9
runtime libraries (Xcode 6.4)

Are you still using:

Python 3.6.13 | packaged by conda-forge | (default, Feb 19 2021, 05:36:16)
[GCC Clang 11.0.1] on darwin

This indicates that Xcode 11 (from macOS 10.15, Catalina) was used.
Newer Xcode versions support for older macOS versions – Xcode 11
could target versions as old as 10.6; Xcode 12 can target version 10.9.

SNAP provides jpy-0.9.0-cp36-cp36m-macosx_10_12_x86_64.whl.

You never mentioned whether you were able to run the simple tests with your previous configuration. The macosx_10_9_x86_64 refers to older macOS runtime libraries. If your previous install never worked even for the simple tests, it is likely that SNAP’s bundled jpy is not compatible with macosx_10_9, so you will need to build jpy as described.

Before trying to build a binary wheel, have a look at Building a Platform Wheel.

Anaconda also has instructions for building binary wheels – you may need to install some conda packages.

#9

Okay, thanks again for all your help.

#10

Dear OlivierLej:

I would like to know if you have finally run snappy on mac OS.
I am trying to run it on MacBookPro mac OSCatalina version 10.15.6.

Kind regards

Marcela

#11

Hi Marcela,

I am able to run the software on my laptop but I’m unable to link it up with Python. I gave up after trying really hard for a couple of days. I’ll try again in a few weeks. In the meantime, if you find a solution, I’d be very keen to hear it.

All the best,
Olivier

#12

If you are able to install and run docker, then I suggest to use a pre-prepared Linux Container with SNAP, SNAPPY (and eventually Jupyter) installed…

Something like:

hope this helps…

#13

@MPereira: Olivier is using MacOS Big Sur, so a step further away from the system used to create the bundled jpy binary wheel. It would be best if you can find a way to use a non-Apple Python-3.6 with the bundled jpy wheel because that configuration is closest to what is working for and well tested on linux and Windows.

Can you describe what you have tried and attach the snappyutil.log? Did you install a 3rd party python-3.6?

If Python-3.6.8 from Python.org runs on your system it may allow you to use the bundled jpy wheel, but you have to be careful to avoid conflicts with Apple’s Python.
The current Apple documentation says you should always run 3rd party programs by providing the full path, e.g., to use ESA SNAP snappy, always run python using the same path as you provided to snappyutil.py.

#15

OlivierLej , gnwiii:

I was applying Anaconda´s python 3.8.5 so according to your recommendation, now I´m going to install a python 3.6 from Python.org.

#16

I think Python.org’s Python-3.6 will provide /usr/local/bin/python3, so in a terminal, /usr/local/bin/python3 to verify that you are getting a 3.6 version and that it actually runs on your more recent macOS version, then change the working directory to the <snap_install_dir>/bin and run .\snappyconf.py /usr/local/bin/python3. If that works, you can continue with the checks from Configure Python to use the SNAP-Python snappy interface, but always run python3 using the full path (e.g., /usr/local/bin/python3).

See: Apple Shell Scripting Command Line Primer.
For macOS the PATH is one of the “Special Shell Variables”, t “A colon-delimited list of locations where trusted executables are installed. Any executable in one of these locations can be executed without specifying a complete path.” (note that many web documents suggest changing the PATH, so use caution).

If the Python.org python-3.6 doesn’t run on your version of macOS or ESA SNAP snappy still can’t be configured, you may be able to install python-3.6 using third party packages such as macports or homebrew. Anaconda python’s older versions have been problematic because they don’t update them for newer versions of macOS. The third party packages may be a better option if you are planning to make heavy use of python with ESA SNAP snappy and expect to need additional python libraries.

#17

I have reinstalled SNAP 8 and installed a Python version 3.6. I have chosen to configure my Python during the installation (I have selected /usr/local/bin/python3)
Then I have changed my directory to: pereira@Marcelas-MacBook-Pro bin % ./snappy-conf /usr/local/bin/python3 but it does not work.
It is the retrieval from snappyutil.log:

Screenshot 2021-04-30 at 14.31.30
Screenshot 2021-04-30 at 14.31.301376×428 128 KB

#18

but snappyutil.log mentions macosx_10_9. The bundled jpy binary wheels are:

% jar tf /home/gwhite/snap/snap/modules/org-esa-snap-snap-python.jar | grep 'jpy.*-macos'
lib/jpy-0.9.0-cp27-cp27m-macosx_10_12_x86_64.whl
lib/jpy-0.9.0-cp34-cp34m-macosx_10_12_x86_64.whl
lib/jpy-0.9.0-cp35-cp35m-macosx_10_12_x86_64.whl
lib/jpy-0.9.0-cp36-cp36m-macosx_10_12_x86_64.whl

This indicates the binary wheels was build for macosx_10_12, but it looks like Python.org built python-3.6 using macosx_10_9. This is not surprising, as they want it to run on many macosx versions, but you generally can’t safely run macosx_10_12 packages in python built for 10_9.

You should try the suggested steps to build a binary wheel using your python-3.6. Note that
<version> is 0.9.0 and be sure your run python with the full path:

cd  ~/.snap/snap-python/snappy
unzip jpy-0.9.0.zip
cd jpy-0.9.0
/usr/local/bin/python3 setup.py bdist_wheel

If this succeeds you just need to copy the resulting dist/*.whl into the snappy directory and
run. You may need to install the Python wheel package, a full JDK, and maven.

#19

I have followed your indications and I have obtained it:

Screenshot 2021-05-03 at 08.46.39
Screenshot 2021-05-03 at 08.46.39694×105 6.11 KB

Would I need to install wheel package, a full JDK and maven? If it would be necessary, could you tell me from where?

#20

You need a full version 8 Java Development Kit (JDK) and also Apache maven. There are third party package managers macports, Fink, or homebrew, or you can find individual macOS packages.
If your organization has other macOS users you should ask around to see what others are using. Some sites maintain a local collection of packages.
If you use one of these package managers you may also be able to install a newer build of python-3.6.

If you have not done any Java programming you probably have to install OpenJDK and then maven. After installing OpenJDK you should check that it is working by building a “hello world” example java program.

For an openjdk package: Open Logic should provide openlogic-openjdk-8u292-b10-mac-x64.pkg.

Step-by-step installation of Java JDK on macOS

Then you need to install [Apache Maven].(https://maven.apache.org/install.html).

Step-by-step maven installation on macOS.

#21

I had to install OpenJDK (I did not have any version installed), I checked if it was working following the example java and it was running. Then I have installed Maven. However I have the same result from snappyutil.log:

Screenshot 2021-05-03 at 16.29.13
Screenshot 2021-05-03 at 16.29.131378×534 140 KB

Also I have installed python 3.8 using Homebrew, so I don´t understand where is the problem.