Configure snap-py on MacOS

Good morning,

I am trying to configure SNAP-PY on my computer, following the tutorial : https://senbox.atlassian.net/wiki/spaces/SNAP/pages/50855941/Configure+Python+to+use+the+SNAP-Python+snappy+interface. I read different topic to try to resolve my problem but I am not very familiar with these kind of problem, so I did not manage to solve it.
The different version I used for JDK/Maven/Python :

So, to begin, when I am in /snap/bin I use the command :
./snappy-conf /usr/bin/python3

And here I have a first error :

Python configuration error: Python configuration failed.
Command [/usr/bin/python3 ./snappyutil.py --snap_home /Applications/snap --java_module /Applications/snap/snap/modules/org-esa-snap-snap-python.jar --force --log_file ./snappyutil.log --jvm_max_mem 11G --java_home /Applications/snap/.install4j/jre.bundle/Contents/Home --req_arch x86_64]
failed with return code 10.
Please check the log file '/Users/floriane/.snap/snap-python/snappy/snappyutil.log'.

Ok, => snappyutil.log

INFO: Installing from Java module '/Applications/snap/snap/modules/org-esa-snap-snap-python.jar'
INFO: Installing jpy...
ERROR: The module 'jpy' is required to run snappy, but no binary 'jpy' wheel matching the pattern
'jpy-{version}-cp38-{abi_tag}-macosx_10_14_x86_64.whl' could be found.
You can try to build a 'jpy' wheel yourself, then copy it into
"/Users/floriane/.snap/snap-python/snappy", and then run the configuration again.
Unzip the jpy sources in /Users/floriane/.snap/snap-python/snappy/jpy-<version>.zip, then
  $ cd jpy-<version>
  $ python setup.py bdist_wheel
  $ cp dist/*.whl "/Users/floriane/.snap/snap-python/snappy"
Or get the source code from https://github.com/bcdev/jpy and follow the build instructions:
  $ git clone https://github.com/bcdev/jpy.git
  $ cd jpy
ERROR: Configuration failed with exit code 10

I did the fist step, but it gets stuck here : “python setup.py bdist_wheel”

But, I have a JDK version greater than 1.7.

I try to do :

floriane@MBP-de-Floriane jpy % export JDK_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
floriane@MBP-de-Floriane jpy % export JAVA_HOME=$JDK_HOME                                                            

But nothing append… what should I do ?

Thank you very much for helping me !

You appear to have python 3.7 on a directory in your PATH variable, but used python 3.8 with ./snappy-conf /usr/bin/python3

The screen capture shows python 3.7.7, but omits the path, so we don’t know where this python is installed. The you use /usr/bin/python3 ./snappyutil.py. The log file says:

The -cp38- indicates that /usr/bin/python3 is version 3.8. You need to find the python 3.7 location and use the full path with snappy-conf and any scripts that use snappy. Note that Linux users often told to adjust their PATH variable, but Apple reserves the PATH variable for “system” programs and tells us to use the full path for non-system programs. You may find other programs failing when they find python 3.7 when they expect python 3.8.

2 Likes

Thanks for your answer…
Ok, I change the PATH with Python3.7, and I have the same error … with -cp37 …

ERROR: The module 'jpy' is required to run snappy, but no binary 'jpy' wheel matching the pattern
'jpy-{version}-cp37-{abi_tag}-macosx_10_9_x86_64.whl' could be found.
You can try to build a 'jpy' wheel yourself, then copy it into
"/Users/floriane/.snap/snap-python/snappy", and then run the configuration again.
Unzip the jpy sources in /Users/floriane/.snap/snap-python/snappy/jpy-<version>.zip, then
  $ cd jpy-<version>
  $ python setup.py bdist_wheel
  $ cp dist/*.whl "/Users/floriane/.snap/snap-python/snappy"
Or get the source code from https://github.com/bcdev/jpy and follow the build instructions:
  $ git clone https://github.com/bcdev/jpy.git
  $ cd jpy
ERROR: Configuration failed with exit code 10

Now the problem is probably the macOS version, so you are left with building the binary wheel using python 3.7.

You pointed JDK_HOME to a “jre”. Try leaving off the /jre.

1 Like

Note also the Python-wrapper snapista which calls gpt for the processing so there is no performance degradation.

https://snap-contrib.github.io/snapista/

Thanks ! It is better now !
Obviously, I still have many errors. I am trying to fix it, but it does not work… :face_with_thermometer:

floriane@MBP-de-Floriane jpy % python3 setup.py bdist_wheel                                 
running bdist_wheel
running build
running build_py
copying jpyutil.py -> build/lib.macosx-10.9-x86_64-3.7
running build_ext
building 'jpy' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_module.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_module.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_diag.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_diag.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_verboseexcept.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_verboseexcept.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_conv.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_conv.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_compat.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_compat.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jtype.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jtype.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jarray.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jarray.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jobj.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jobj.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jmethod.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jmethod.o
src/main/c/jpy_jmethod.c:857:5: warning: code will never be executed [-Wunreachable-code]
    PyErr_SetString(PyExc_RuntimeError, "internal error");
    ^~~~~~~~~~~~~~~
1 warning generated.
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jpy_jfield.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jpy_jfield.o
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Isrc/main/c -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/include/darwin -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/main/c/jni/org_jpy_PyLib.c -o build/temp.macosx-10.9-x86_64-3.7/src/main/c/jni/org_jpy_PyLib.o
src/main/c/jni/org_jpy_PyLib.c:167:31: warning: passing 'const wchar_t *' (aka 'const int *') to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
                PyMem_RawFree(pythonHome);
                              ^~~~~~~~~~
/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pymem.h:18:38: note: passing argument to parameter 'ptr' here
PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
                                     ^
src/main/c/jni/org_jpy_PyLib.c:184:27: warning: passing 'const wchar_t *' (aka 'const int *') to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
            PyMem_RawFree(pythonHome);
                          ^~~~~~~~~~
/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pymem.h:18:38: note: passing argument to parameter 'ptr' here
PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
                                     ^
**src/main/c/jni/org_jpy_PyLib.c:331:9: error: implicit declaration of function 'JPy_free' is invalid in C99 [-Werror,-Wimplicit-function-declaration]**
**        JPy_free();**
        ^
src/main/c/jni/org_jpy_PyLib.c:329:26: warning: unused variable 'state' [-Wunused-variable]
        PyGILState_STATE state = PyGILState_Ensure();
                         ^
src/main/c/jni/org_jpy_PyLib.c:791:16: warning: incompatible pointer to integer conversion returning 'void *' from a function with result type 'jlong' (aka 'long') [-Wint-conversion]
        return NULL;
               ^~~~
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/stddef.h:89:16: note: expanded from macro 'NULL'
#  define NULL ((void*)0)
               ^~~~~~~~~~
src/main/c/jni/org_jpy_PyLib.c:796:101: warning: passing 'const char *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
    result = executeInternal(jenv, jLibClass, jStart, jGlobals, jLocals, (DoRun)pyRunStringWrapper, codeChars);
                                                                                                    ^~~~~~~~~
src/main/c/jni/org_jpy_PyLib.c:657:128: note: passing argument to parameter 'runArg' here
jlong executeInternal(JNIEnv* jenv, jclass jLibClass, jint jStart, jobject jGlobals, jobject jLocals, DoRun runFunction, void *runArg) {
                                                                                                                               ^
src/main/c/jni/org_jpy_PyLib.c:1076:14: warning: incompatible pointer types assigning to 'PyObject *' (aka 'struct _object *') from 'struct _typeobject *' [-Wincompatible-pointer-types]
    pyObject = ((PyObject*) objId)->ob_type;
             ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6 warnings and 1 error generated.
error: command 'gcc' failed with exit status 1

The error here is : src/main/c/jni/org_jpy_PyLib.c:331:9: error: implicit declaration of function ‘JPy_free’ is invalid in C99 [-Werror,-Wimplicit-function-declaration]

GCC and XCode are well installed. Maybe do you have an idea to resolve this problem ? :worried:
Thank you very much !!

Generally, macOS software uses clang, not gcc. If you have the CC=gcc in your environment or python built using gcc that could explain why gcc is used by setup.py. You should check which compilers were used to build your JDK and python 3.7 (this should be mentioned in the documentation (OpenJDK Build Instructions say clang is used for on macOS), but may not mention versions. Apple’s otool will show you which libraries are linked by python and java’s jvm. Third party software is often built on older macOS versions and will run on newer versions but can have problems if you then build extensions on the newer system.

1 Like

I fixed the problem !!! It works !!! Thank you very very very much !!! I am able to import snappy :smiley: !! Really, thank you !!

1 Like

Glad you got it working.

It might be helpful to others if you can post a summary of the key steps needed to configure ESA SNAP snappy on your macOS version. From the information in this thread, it seems the instructions SNAP provides (when snappy-util.py fails and the README.md file that comes with jpy) are correct, but assume the reader is an experienced Python and Java developer on their platform. To me it seems the key stumbling points were ensuring that setup.py was run with python 3.7 then I’m guessing you needed to get setup.py to use Apple’s clang compiler.

1 Like

The different steps, I used to resolve this problem was :

  1. I had to choose the appropriate PATH from Python 3.7, when I launched the command line

./snappy-conf <python-PATH>

(Or at least, the one you use => you can check it with python3 --version)

If the ‘JPY’ module is not found with your python version, you can get the source code with this command line :

git clone https://github.com/bcdev/jpy.git
cd jpy

  1. After that, if you type :

python setup.py bdist_wheel

and you have an error as : "JAVA_HOME" must be set to a JDK (>= v1.7) installation directory), you have to export the PATH of JDK as :

export JDK_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/

(or your path) and,

export JAVA_HOME=$JDK_HOME

  1. If the error is linked to CLANG and GCC, you could resolve it, with this command line :

export CC=/usr/local/bin/gdc

After that, it should be work! Now you could type :

python setup.py bdist_wheel
cp dist/*.whl “/Users/floriane/.snap/snap-python/snappy”
./snappy-conf <python3-PATH>
cd <snappy-dir>

and the following commands are : https://senbox.atlassian.net/wiki/spaces/SNAP/pages/50855941/Configure+Python+to+use+the+SNAP-Python+snappy+interface

Hope it is enough clear…

1 Like

It would be good to mention where you got Python 3.7 (e.g., Apple, Python.org, macports, homebrew, anaconda, or ??) as this may affect the choice of compiler (clang versus gcc).

I assume this is a typo for export CC=/usr/local/bin/gcc.

Python3.7 comes from Python.org, I don’t use anaconda. Libraries are installed with pip3 or homebrew.

YEs, sorry for the typo !