Snappy doesn't clear memory cache

development
5

Hi,
I’m encountering the same issue: snappy does not clear the memory cash. The code below is a simple example, similar to that shared by atteggiani, which illustrates the problem: I simply loop through S1 products and plot subsets of the intensity bands to png files. I always experience this issue, no matter the complexity of processing pipeline, and no matter the product type (S1 or S2).

Several posts indicate how to tweak the various memory settings (post) and tile cache size (post), however I would like to know how to properly release the memory, which is the only way to process large number of files sequentially.

I have tried various approaches discussed elsewhere, with no success:
(1) dispose the product to release all resources used by the object (post)
(2) explicitly call the Java Garbage Collection (post)
(3) try to restart the Java Virtual Machine (post), I obtain the same error described by atteggiani

import os
import sys
import time
import snappy
from snappy import ProductIO
from snappy import HashMap
from snappy import GPF
from snappy import jpy

# --- MEMORY CONFIGURATION:
# (http://forum.step.esa.int/t/snappy-error-productio-writeproduct/1102)
# pico /usr/local/lib/python2.7/dist-packages/snappy/snappy.ini
#   java_max_mem: 15G
# pico ~/.snap/snap-python/snappy/jpyconfig.py
#   jvm_maxmem = '15G'
#
# --- TILE CACHE SIZE CONFIGURATION for speed
# (http://forum.step.esa.int/t/slower-snappy-processing/6354/7)
# pico ~/snap/etc/snap.properties
#   snap.jai.tileCacheSize = 10000

JAI = jpy.get_type('javax.media.jai.JAI')
ImageManager = jpy.get_type('org.esa.snap.core.image.ImageManager')
System = jpy.get_type('java.lang.System')
System.setProperty('com.sun.media.jai.disableMediaLib', 'true')

zipfiles = ['S1B_IW_SLC__1SDV_20171020T152624_20171020T152654_007915_00DFA8_DD8E.zip',
            'S1B_IW_SLC__1SDV_20171101T152624_20171101T152654_008090_00E4B0_3C6D.zip',
            'S1B_IW_SLC__1SDV_20171113T152624_20171113T152654_008265_00E9E0_8385.zip',
            'S1B_IW_SLC__1SDV_20171125T152621_20171125T152651_008440_00EF32_AB25.zip',
            ]

for f in zipfiles:
    # --- read product
    p = ProductIO.readProduct(f)

    # --- apply orbit file
    parameters = HashMap()
    p = GPF.createProduct('Apply-Orbit-File', parameters, p)

    # --- deburst
    parameters = HashMap()
    p = GPF.createProduct('TOPSAR-Deburst', parameters, p)

    # --- terrain correction
    sourcebands = ['Intensity_VH', 'Intensity_VV']
    sourceBands_str = ','.join(sourcebands)
    parameters = HashMap()
    parameters.put('sourceBands', sourceBands_str)
    p = GPF.createProduct('Terrain-Correction', parameters, p)

    # --- subset
    geoRegion = 'POLYGON((40.63 13.64, 40.735 13.64, 40.735 13.53, 40.63 13.53, 40.63 13.64))'
    parameters = HashMap()
    parameters.put('copyMetadata', 'true')
    parameters.put('geoRegion', geoRegion)
    p = GPF.createProduct('Subset', parameters, p)

    # --- get metadata
    acqstart = p.getMetadataRoot().getElement('Abstracted_Metadata').getAttributeString('first_line_time')

    # --- plot bands
    # https://github.com/senbox-org/snap-engine/blob/9b7bffd69e18f71177afaa45c2e37468959e29b8/snap-python/src/main/resources/snappy/examples/snappy_write_image.py
    for bname in sourcebands:
        print('- plotting band "' + bname + '"')
        band = p.getBand(bname)
        im = ImageManager.getInstance().createColoredBandImage([band], band.getImageInfo(), 0)
        f_out = '{}_{}.png'.format(acqstart[0:10], bname)
        JAI.create("filestore", im, f_out, 'png')

    # --- FREE MEMORY
    # - attempt 1: DISPOSE product
    print('- dispose product')
    p.dispose()
    time.sleep(2)

    # - attempt 2: GARBAGE COLLECTOR
    # http://forum.step.esa.int/t/how-to-free-java-memory-snappy/5738
    System = jpy.get_type('java.lang.System')
    System.gc()
    time.sleep(2)

    # - attempt 3: JVM restart
    # http://forum.step.esa.int/t/snappy-doesnt-clear-memory-cache/8284/3
    jpy.destroy_jvm()
    time.sleep(2)
    print('- create JVM')
    opt = snappy.jpyutil.get_jvm_options()
    jpy.create_jvm(opt)
    time.sleep(2)

Am I doing something wrong? Are there any recommendations on how to proceed?

Many thanks in advance,
Sébastien

1 Like