SNAPPY install on a high performance cluster: install failure and security defect

development python
#1

I and our IT team are trying to get SNAPPY working on a high-performance cluster (Linux). The plan is to use a common SNAP install to supply the Python bindings to each user who wants them. However, we have run into a number of issues as detailed below. I guess that ESA or another institution has this problem solved? If so it would be great to share what the solution is. I copy below the feedback from our scientific computing lead for context as to the issues.

+++
(1) Snappy-conf fails when trying to request a large amount of memory on our system
… --jvm_max_mem 176G

(2) Any attempt to bypass this issue either by editing snappyutil.py or setting java_max_mem in snappy.ini fails, because both snappyutil.py and snappy.ini are re-downloaded for every run of snappy-conf. So any changes made are lost.

(3) Finally, if you manually extract the install line, and then manually change the --jvm_max_mem setting, you can get snappy to install. However, it gives the following error when we import the snappy module:
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH does not contain the current folder ‘.’. Its value is ‘/cm/local/apps/torque/6.1.3/lib’.

Most importantly: adding . to LD_LIBRARY_PATH is bad practice. Firstly LD_LIBRARY_PATH is used by all binary programs in your environment, every single time they are invoked. And secondly, means the current directory, which changes as you move around the filesystem. All it takes is for some malicious user to create a doctored copy of, e.g., libm.so in a non-standard location and you may find yourself loading the doctored version of libm.so rather than the system one. It’s generally considered bad practice to use .LD_LIBRARY_PATH anywhere.
+++

I have seen some other users getting around such issues by loading individual toolboxes into Docker containers, but Docker is not useable on the system we have.

Any help- or direct contacts to email on this subject would be much appreciated!

#2

Hello!

(1) What’s the error message? I can’t image a reason why it could fail.

(2) Sure, when re-configuring snappy the defaults are restored.

Regarding point (3) I can’t say much, not that I was very helpful for point (1) and (2) till now.
Maybe @oana_hogoiu can say something regarding point (3).

#3

The first thing that pops out is the location of this class, in your error message. Now it’s located under org.esa.s2tbx.dataio.gdal.GDALDistributionInstaller, so it seems that your SNAP version is old. Also, I searched after your error on the forum and I only found it in old posts.

#5

Hi both,

Apologies for the delay in response, it took a while for the IT team to get back around to it. Oana was entirely correct, the issue was driven by the attempt to use an old version of Snap, the new version has worked. There were still some issues with the installer requesting a huge amount of memory space, beyond that available on one node, but a solution was able to be found to that.

Many thanks for the assistance.