Configure snap-py on MacOS

Good morning,

I am trying to configure SNAP-PY on my computer, following the tutorial : https://senbox.atlassian.net/wiki/spaces/SNAP/pages/50855941/Configure+Python+to+use+the+SNAP-Python+snappy+interface. I read different topic to try to resolve my problem but I am not very familiar with these kind of problem, so I did not manage to solve it.
The different version I used for JDK/Maven/Python :

So, to begin, when I am in /snap/bin I use the command :
./snappy-conf /usr/bin/python3

And here I have a first error :

Python configuration error: Python configuration failed.
Command [/usr/bin/python3 ./snappyutil.py --snap_home /Applications/snap --java_module /Applications/snap/snap/modules/org-esa-snap-snap-python.jar --force --log_file ./snappyutil.log --jvm_max_mem 11G --java_home /Applications/snap/.install4j/jre.bundle/Contents/Home --req_arch x86_64]
failed with return code 10.
Please check the log file '/Users/floriane/.snap/snap-python/snappy/snappyutil.log'.

Ok, => snappyutil.log

INFO: Installing from Java module '/Applications/snap/snap/modules/org-esa-snap-snap-python.jar'
INFO: Installing jpy...
ERROR: The module 'jpy' is required to run snappy, but no binary 'jpy' wheel matching the pattern
'jpy-{version}-cp38-{abi_tag}-macosx_10_14_x86_64.whl' could be found.
You can try to build a 'jpy' wheel yourself, then copy it into
"/Users/floriane/.snap/snap-python/snappy", and then run the configuration again.
Unzip the jpy sources in /Users/floriane/.snap/snap-python/snappy/jpy-<version>.zip, then
  $ cd jpy-<version>
  $ python setup.py bdist_wheel
  $ cp dist/*.whl "/Users/floriane/.snap/snap-python/snappy"
Or get the source code from https://github.com/bcdev/jpy and follow the build instructions:
  $ git clone https://github.com/bcdev/jpy.git
  $ cd jpy
ERROR: Configuration failed with exit code 10

I did the fist step, but it gets stuck here : “python setup.py bdist_wheel”

But, I have a JDK version greater than 1.7.

I try to do :

floriane@MBP-de-Floriane jpy % export JDK_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
floriane@MBP-de-Floriane jpy % export JAVA_HOME=$JDK_HOME                                                            

But nothing append… what should I do ?

Thank you very much for helping me !

You appear to have python 3.7 on a directory in your PATH variable, but used python 3.8 with ./snappy-conf /usr/bin/python3

The screen capture shows python 3.7.7, but omits the path, so we don’t know where this python is installed. The you use /usr/bin/python3 ./snappyutil.py. The log file says:

The -cp38- indicates that /usr/bin/python3 is version 3.8. You need to find the python 3.7 location and use the full path with snappy-conf and any scripts that use snappy. Note that Linux users often told to adjust their PATH variable, but Apple reserves the PATH variable for “system” programs and tells us to use the full path for non-system programs. You may find other programs failing when they find python 3.7 when they expect python 3.8.

2 Likes

Thanks for your answer…
Ok, I change the PATH with Python3.7, and I have the same error … with -cp37 …

ERROR: The module 'jpy' is required to run snappy, but no binary 'jpy' wheel matching the pattern
'jpy-{version}-cp37-{abi_tag}-macosx_10_9_x86_64.whl' could be found.
You can try to build a 'jpy' wheel yourself, then copy it into
"/Users/floriane/.snap/snap-python/snappy", and then run the configuration again.
Unzip the jpy sources in /Users/floriane/.snap/snap-python/snappy/jpy-<version>.zip, then
  $ cd jpy-<version>
  $ python setup.py bdist_wheel
  $ cp dist/*.whl "/Users/floriane/.snap/snap-python/snappy"
Or get the source code from https://github.com/bcdev/jpy and follow the build instructions:
  $ git clone https://github.com/bcdev/jpy.git
  $ cd jpy
ERROR: Configuration failed with exit code 10

Now the problem is probably the macOS version, so you are left with building the binary wheel using python 3.7.

You pointed JDK_HOME to a “jre”. Try leaving off the /jre.

1 Like

Note also the Python-wrapper snapista which calls gpt for the processing so there is no performance degradation.

https://snap-contrib.github.io/snapista/

Thanks ! It is better now !
Obviously, I still have many errors. I am trying to fix it, but it does not work… :face_with_thermometer:

floriane@MBP-de-Floriane jpy % python3 setup.py bdist_wheel                                 
running bdist_wheel
running build
running build_py
copying jpyutil.py -> build/lib.macosx-10.9-x86_64-3.7
running build_ext
building 'jpy' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_module.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_module.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_diag.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_diag.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_verboseexcept.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_verboseexcept.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_conv.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_conv.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_compat.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_compat.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jtype.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jtype.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jarray.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jarray.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jobj.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jobj.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jmethod.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jmethod.o
src/main/c/jpy_jmethod.c:857:5: warning: code will never be executed [-Wunreachable-code]
    PyErr_SetString(PyExc_RuntimeError, "internal error");
    ^~~~~~~~~~~~~~~
1 warning generated.
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jfield.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jfield.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jni/org_jpy_PyLib.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jni/org_jpy_PyLib.o
src/main/c/jni/org_jpy_PyLib.c:167:31: warning: passing 'const wchar_t *' (aka 'const int *') to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
                PyMem_RawFree(pythonHome);
                              ^~~~~~~~~~
/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pymem.h:18:38: note: passing argument to parameter 'ptr' here
PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
                                     ^
src/main/c/jni/org_jpy_PyLib.c:184:27: warning: passing 'const wchar_t *' (aka 'const int *') to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
            PyMem_RawFree(pythonHome);
                          ^~~~~~~~~~
/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pymem.h:18:38: note: passing argument to parameter 'ptr' here
PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
                                     ^
**src/main/c/jni/org_jpy_PyLib.c:331:9: error: implicit declaration of function 'JPy_free' is invalid in C99 [-Werror,-Wimplicit-function-declaration]**
**        JPy_free();**
        ^
src/main/c/jni/org_jpy_PyLib.c:329:26: warning: unused variable 'state' [-Wunused-variable]
        PyGILState_STATE state = PyGILState_Ensure();
                         ^
src/main/c/jni/org_jpy_PyLib.c:791:16: warning: incompatible pointer to integer conversion returning 'void *' from a function with result type 'jlong' (aka 'long') [-Wint-conversion]
        return NULL;
               ^~~~
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/stddef.h:89:16: note: expanded from macro 'NULL'
#  define NULL ((void*)0)
               ^~~~~~~~~~
src/main/c/jni/org_jpy_PyLib.c:796:101: warning: passing 'const char *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
    result = executeInternal(jenv, jLibClass, jStart, jGlobals, jLocals, (DoRun)pyRunStringWrapper, codeChars);
                                                                                                    ^~~~~~~~~
src/main/c/jni/org_jpy_PyLib.c:657:128: note: passing argument to parameter 'runArg' here
jlong executeInternal(JNIEnv* jenv, jclass jLibClass, jint jStart, jobject jGlobals, jobject jLocals, DoRun runFunction, void *runArg) {
                                                                                                                               ^
src/main/c/jni/org_jpy_PyLib.c:1076:14: warning: incompatible pointer types assigning to 'PyObject *' (aka 'struct _object *') from 'struct _typeobject *' [-Wincompatible-pointer-types]
    pyObject = ((PyObject*) objId)->ob_type;
             ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6 warnings and 1 error generated.
error: command 'gcc' failed with exit status 1

The error here is : src/main/c/jni/org_jpy_PyLib.c:331:9: error: implicit declaration of function ‘JPy_free’ is invalid in C99 [-Werror,-Wimplicit-function-declaration]

GCC and XCode are well installed. Maybe do you have an idea to resolve this problem ? :worried:
Thank you very much !!

Generally, macOS software uses clang, not gcc. If you have the CC=gcc in your environment or python built using gcc that could explain why gcc is used by setup.py. You should check which compilers were used to build your JDK and python 3.7 (this should be mentioned in the documentation (OpenJDK Build Instructions say clang is used for on macOS), but may not mention versions. Apple’s otool will show you which libraries are linked by python and java’s jvm. Third party software is often built on older macOS versions and will run on newer versions but can have problems if you then build extensions on the newer system.

1 Like

I fixed the problem !!! It works !!! Thank you very very very much !!! I am able to import snappy :smiley: !! Really, thank you !!

1 Like

Glad you got it working.

It might be helpful to others if you can post a summary of the key steps needed to configure ESA SNAP snappy on your macOS version. From the information in this thread, it seems the instructions SNAP provides (when snappy-util.py fails and the README.md file that comes with jpy) are correct, but assume the reader is an experienced Python and Java developer on their platform. To me it seems the key stumbling points were ensuring that setup.py was run with python 3.7 then I’m guessing you needed to get setup.py to use Apple’s clang compiler.

1 Like

The different steps, I used to resolve this problem was :

  1. I had to choose the appropriate PATH from Python 3.7, when I launched the command line

./snappy-conf <python-PATH>

(Or at least, the one you use => you can check it with python3 --version)

If the ‘JPY’ module is not found with your python version, you can get the source code with this command line :

git clone https://github.com/bcdev/jpy.git
cd jpy

  1. After that, if you type :

python setup.py bdist_wheel

and you have an error as : "JAVA_HOME" must be set to a JDK (>= v1.7) installation directory), you have to export the PATH of JDK as :

export JDK_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/

(or your path) and,

export JAVA_HOME=$JDK_HOME

  1. If the error is linked to CLANG and GCC, you could resolve it, with this command line :

export CC=/usr/local/bin/gdc

After that, it should be work! Now you could type :

python setup.py bdist_wheel
cp dist/*.whl “/Users/floriane/.snap/snap-python/snappy”
./snappy-conf <python3-PATH>
cd <snappy-dir>

and the following commands are : https://senbox.atlassian.net/wiki/spaces/SNAP/pages/50855941/Configure+Python+to+use+the+SNAP-Python+snappy+interface

Hope it is enough clear…

1 Like

It would be good to mention where you got Python 3.7 (e.g., Apple, Python.org, macports, homebrew, anaconda, or ??) as this may affect the choice of compiler (clang versus gcc).

I assume this is a typo for export CC=/usr/local/bin/gcc.

Python3.7 comes from Python.org, I don’t use anaconda. Libraries are installed with pip3 or homebrew.

YEs, sorry for the typo !

I had the same issue. Since I don’t have gcc installed, I added the line void JPy_free(); to the top of src/main/c/jni/org_jpy_PyLib.c, and then clang compiled correctly. Note: I used anaconda python, and some things with brew.

Note: make sure to run the python setup.py install command.

Some notes on a further issue I had that I want to dump in case it’s helpful to anyone.

After I installed when trying to import the library, I got the following output:
`Python 3.6.13 | packaged by conda-forge | (default, Sep 23 2021, 07:55:15)
[GCC Clang 11.1.0] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.

import snappy
No Java runtime present, requesting install.`

I was using openjdk15. The most important change seemed to be switching to openjdk8 (see How can I install java openjdk 8 on High Sierra or newer macOS - Ask Different). Remember to update your java environment paths, and I reran the install command to be safe.

Some other things I did that may or may not have helped:
I added m.dylib as an option as in Snappy installation on OS.X Sierra failing (or asking for legacy Java SE 6) - #2 by forman

I also removed the jre paths in jpyconfig.py.

Hello! I have the same issue with the gcc error.

I have Python 3.6.8 from Python.org, an M1 Mac with Big Sur 11.2.2 & SNAP 8.0

I tried the CC=/usr/local/bin/gcc but that gave a new set of error:
unable to execute ‘/usr/local/bin/gcc’: No such file or directory
error: command ‘/usr/local/bin/gcc’ failed with exit status 1
So I looked in my /usr/local/bin folder and I don’t have a gcc file but I do have a gcc-11 but that didn’t work
Then I tried this video to make gcc-11 (homebrew gcc) be the main gcc (instead of clang gcc) as per this youtube video but that didn’t work either Install GCC/G++ on MacOS arm64 M1 Mac - YouTube

I added void JPy_free(); to the top of src/main/c/jni/org_jpy_PyLib.c
and I still get the gcc error

I feel like I’ve tried almost all the solutions proposed but I’m not sure what else to try or maybe if I tried too many solutions at the same time? Any help would be greatly appreciated.

My goal is to use snappy with other packages in python to automate some sentinel-1 snap processing.

Using the M1 processor make your job more complicated. The SNAP software currently uses Intel binaries, so runs under rosetta. Both Python and jpy need to be built for Intel architecture with compatible compiler versions. There are reports of failures with Python 3.6.8 from Python.org on recent MacOS versions because Python 3.6.8 was built on a much older macOS version.

Building a Universal MacOS Binary shows how to build Intel binaries using X-code (universal binaries are created by combining Intel and M1 binaries, but you just need Intel binaries).

You will probably need to use gcc, but you may need to specifically install a version that generates Intel
binaries (some may only generate M1 binaries).

You don’t mention your JDK version. You should install Intel macOS openjdk-8.jdk as mentioned in previous posts. There are many sources for gcc and openjdk-8.jdk, so you will have to check the one you use to make sure it can generate Intel binaries (there may be arguments to select the target architecture).