The crash happened outside the Java Virtual Machine in native code - exception when processing multiple files using snappy

development python
#1

I use snappy Python module to process Sentinel-1 data. My code is at https://github.com/kedziorm/mySNAPscripts/blob/master/myScripts.py

When I process only one file (run ipython, paste mentioned Python script, call appropriate functions) I have no problems. However if I try to process multiple files, I finally get an exception.
I’ve thought that some objects might not be removed from the memory, so I set them to None:

I’ve also set ulimit -c unlimited as suggested in exception:

I still receive the same error message as previously (so I’m pasting previous error message).
How can I fix that?

ipython_Java_exception.png557×877 88.4 KB

hs_err_pid9947.log (182,3 KB)

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000000055fcfb, pid=9947, tid=140521561863936
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [python+0x15fcfb]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# ~/hs_err_pid9947.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#2

Hi,

I had this error when trying to open SNAP.

Remove the org-esa-snap-snap-worldwind.jar from the snap folder and back up it in an other folder on your computer.

Should work.

#3

I’ve no idea where this error might come from. But it’s not the world-wind module.
Looking at the error-log I would think it is somewhere in the native python code.
In the screenshot ipython 2.4.1-1 is mentioned. Maybe the problem is there. I don’t want to say that SNAP/snappy might not be guilty but I can’t see it.

#4

Could you suggest any idea of trying to find a source of the issues?

Should I start a “raw” Python (just typing python and execute all the code within it?) instead of using ipython?
Should I ask about ‘ipython’ memory issues on http://stackoverflow.com (I’m not sure if it is causing this issue, so I would like to ensure).

#5

Thanks Sadri,

I have the same problem as reported here when processing a series of S2 data. At each product this bit seemed to cause problems:

    GPF.getDefaultInstance().getOperatorSpiRegistry().loadOperatorSpis()

    HashMap = jpy.get_type('java.util.HashMap')
    BandDescriptor = jpy.get_type('org.esa.snap.core.gpf.common.BandMathsOp$BandDescriptor')

    parameters = HashMap()
    parameters.put('targetResolution', samp_band)
    parameters.put('upsampling', 'Bilinear')
    parameters.put('downsampling', 'Mean')
    parameters.put('flagDownsampling', 'FlagMedianAnd')
    parameters.put('resampleOnPyramidLevels', True)

    product = GPF.createProduct('Resample', parameters, self.source_product)
    source_angle = product.getBand(band_angle)

I have tried different tricks as the one posted here (garbage collector, increase virtual memory…). I have also monitored the memory consumption and found not to increase.
I have also moved the file “org-esa-snap-snap-worldwind.jar” as you suggested and the execution did not work yet. My assumption is that the JVM does not release memory correctly but I cannot see the reason why. I try to “dispose” and clear memory but does seem not to work

Thanks

Javier Gorrono

#6

Following these comments, I have found a “dirty” workaround to the problem.

The trick to avoid this JVM crash has been solved in my case by externally managing the program with a bash command.
#! /bin/bash
for initprod in seq 0 64;
do
python run.py $initprod $((++initprod))
done

By doing so, it seems that I can run as many products as I want since the JVM is initialised and cleared at every iteration.
It makes the programme structure a bit more tedious but it is a current limitation of jpy library since, it cannot “shutdown the Java VM from Python and then restart it”.

Cheers

Javier Gorrono

2 Likes
#7

From my experience this crash is caused by calling jpy.get_type multiple times. Only call it once at the beginning and your crash should be solved. See this thread.

2 Likes