Snappy doesn't clear memory cache

[seb]

Hi,
I’m encountering the same issue: snappy does not clear the memory cash. The code below is a simple example, similar to that shared by atteggiani, which illustrates the problem: I simply loop through S1 products and plot subsets of the intensity bands to png files. I always experience this issue, no matter the complexity of processing pipeline, and no matter the product type (S1 or S2).

Several posts indicate how to tweak the various memory settings (post) and tile cache size (post), however I would like to know how to properly release the memory, which is the only way to process large number of files sequentially.

I have tried various approaches discussed elsewhere, with no success:
(1) dispose the product to release all resources used by the object (post)
(2) explicitly call the Java Garbage Collection (post)
(3) try to restart the Java Virtual Machine (post), I obtain the same error described by atteggiani

import os
import sys
import time
import snappy
from snappy import ProductIO
from snappy import HashMap
from snappy import GPF
from snappy import jpy

# --- MEMORY CONFIGURATION:
# (http://forum.step.esa.int/t/snappy-error-productio-writeproduct/1102)
# pico /usr/local/lib/python2.7/dist-packages/snappy/snappy.ini
#   java_max_mem: 15G
# pico ~/.snap/snap-python/snappy/jpyconfig.py
#   jvm_maxmem = '15G'
#
# --- TILE CACHE SIZE CONFIGURATION for speed
# (http://forum.step.esa.int/t/slower-snappy-processing/6354/7)
# pico ~/snap/etc/snap.properties
#   snap.jai.tileCacheSize = 10000

JAI = jpy.get_type('javax.media.jai.JAI')
ImageManager = jpy.get_type('org.esa.snap.core.image.ImageManager')
System = jpy.get_type('java.lang.System')
System.setProperty('com.sun.media.jai.disableMediaLib', 'true')

zipfiles = ['S1B_IW_SLC__1SDV_20171020T152624_20171020T152654_007915_00DFA8_DD8E.zip',
            'S1B_IW_SLC__1SDV_20171101T152624_20171101T152654_008090_00E4B0_3C6D.zip',
            'S1B_IW_SLC__1SDV_20171113T152624_20171113T152654_008265_00E9E0_8385.zip',
            'S1B_IW_SLC__1SDV_20171125T152621_20171125T152651_008440_00EF32_AB25.zip',
            ]

for f in zipfiles:
    # --- read product
    p = ProductIO.readProduct(f)

    # --- apply orbit file
    parameters = HashMap()
    p = GPF.createProduct('Apply-Orbit-File', parameters, p)

    # --- deburst
    parameters = HashMap()
    p = GPF.createProduct('TOPSAR-Deburst', parameters, p)

    # --- terrain correction
    sourcebands = ['Intensity_VH', 'Intensity_VV']
    sourceBands_str = ','.join(sourcebands)
    parameters = HashMap()
    parameters.put('sourceBands', sourceBands_str)
    p = GPF.createProduct('Terrain-Correction', parameters, p)

    # --- subset
    geoRegion = 'POLYGON((40.63 13.64, 40.735 13.64, 40.735 13.53, 40.63 13.53, 40.63 13.64))'
    parameters = HashMap()
    parameters.put('copyMetadata', 'true')
    parameters.put('geoRegion', geoRegion)
    p = GPF.createProduct('Subset', parameters, p)

    # --- get metadata
    acqstart = p.getMetadataRoot().getElement('Abstracted_Metadata').getAttributeString('first_line_time')

    # --- plot bands
    # https://github.com/senbox-org/snap-engine/blob/9b7bffd69e18f71177afaa45c2e37468959e29b8/snap-python/src/main/resources/snappy/examples/snappy_write_image.py
    for bname in sourcebands:
        print('- plotting band "' + bname + '"')
        band = p.getBand(bname)
        im = ImageManager.getInstance().createColoredBandImage([band], band.getImageInfo(), 0)
        f_out = '{}_{}.png'.format(acqstart[0:10], bname)
        JAI.create("filestore", im, f_out, 'png')

    # --- FREE MEMORY
    # - attempt 1: DISPOSE product
    print('- dispose product')
    p.dispose()
    time.sleep(2)

    # - attempt 2: GARBAGE COLLECTOR
    # http://forum.step.esa.int/t/how-to-free-java-memory-snappy/5738
    System = jpy.get_type('java.lang.System')
    System.gc()
    time.sleep(2)

    # - attempt 3: JVM restart
    # http://forum.step.esa.int/t/snappy-doesnt-clear-memory-cache/8284/3
    jpy.destroy_jvm()
    time.sleep(2)
    print('- create JVM')
    opt = snappy.jpyutil.get_jvm_options()
    jpy.create_jvm(opt)
    time.sleep(2)

Am I doing something wrong? Are there any recommendations on how to proceed?

Many thanks in advance,
Sébastien

[Ciaran_Evans]

Second this post, there doesn’t seem to be a easy way to force this to happen. My API is rendered useless after a few calls: http://forum.step.esa.int/t/snappy-not-freeing-memory/9274

[Ciaran_Evans]

Sorry to keep bringing this one up, is there any other suggestions of ways to clear this cache? It seems each method in the posts above is not effective.

Appreciate the help a lot!

[pedroLanz]

same issue here–

[gbaier]

bump. I’m having the same issue of running out of memory when using ProductIO.writeProduct in a loop.

[geoagr2003]

Do you have something similar to this situation ?

Using StatisticsOp operator with Python

Hi, I’m trying to do some zonal statistics over the cloud confidence band from Sentinel-2 by using the StatisticsOp operator. I have the following code: class CheckClouds: def __init__(self, sat_img_path): self.sat_img_path = sat_img_path def read_cloud_band(self): ''' Read the cloud confidence band. ''' product = ProductIO.readProduct(self.sat_img_path) band_names = product.getBandNames() # print (list(band_names)) b_qcc = product.getBand('quality_cloud_confiden…

I’m trying to loop several products as well and the loop stops unexpected without error.

[marpet]

I’ve just replied to another post:

Snappy not freeing memory

No there is no special ticket for memory issue in snappy, but there are some others https://senbox.atlassian.net/browse/SNAP-869 https://senbox.atlassian.net/browse/SITBX-562 https://senbox.atlassian.net/browse/SITBX-482 I’ve created one now. (SNAP-960). For me it is not yet clear where the memory issues comes from. Maybe it is within SNAP, maybe in jpy or in snappy. This needs to be investigated. However, I can tell that the further development of SNAP and the Toolboxes will start again…

So you can expect progress soon.

[Ciaran_Evans]

@gbaier and @geoagr2003 it is worth saying that post that @marpet linked also has a workaround that will unblock your development for now whilst the issue is addressed by the team!

[geoagr2003]

You mean the one with around the getting out of the loop jpy.get_type() ?
Do you know if the operator parameters can be configured as nested dictionaries in the current version of snappy ?

[Ciaran_Evans]

No sorry, it’s the issue where running multiple processing jobs inevitably means you’ll run out of memory.

Do you have an example of what you’re trying to do with the operator parameters?

[geoagr2003]

Here is my piece of code:

Using StatisticsOp operator with Python

Hi, I’m trying to do some zonal statistics over the cloud confidence band from Sentinel-2 by using the StatisticsOp operator. I have the following code: class CheckClouds: def __init__(self, sat_img_path): self.sat_img_path = sat_img_path def read_cloud_band(self): ''' Read the cloud confidence band. ''' product = ProductIO.readProduct(self.sat_img_path) band_names = product.getBandNames() # print (list(band_names)) b_qcc = product.getBand('quality_cloud_confiden…

I believe that to many class instantiation happens within the loop and this generated objects won’t destroy after usage.
What do you think ?

[Ciaran_Evans]

It’s a bit of everything really. Any time you run a loop with snappy, anything you instantiate or perform in memory is not garbage collected.

For your example, I’d use my work around and run read_cloud_band()in a separate script. You’ll find that each time the script terminates the memory will be freed.

This is taken from my workaround post:

The line of code I use to spawn my processing pipeline is:
pipeline_out = subprocess.check_output(['python', 'src/SarPipeline.py', location_wkt], stderr=subprocess.STDOUT)

Note: pipeline_out is the STDOUT from the script, so in my case to find out what file has just been processed I have print("filepath: " + path_to_file) in src/SarPipeline.py

So for your code I would try

for file_name in file_list:
    print (file_name)
    pipeline_out = subprocess.check_output(['python', 'readCloudBand.py',file_name], stderr=subprocess.STDOUT)
    #Your old call CheckClouds(file_name).read_cloud_band()

Where readCloudBand.py is the method in your CheckClouds class in a python script and file_name is a passed in parameter to the script.

[Ciaran_Evans]

Subprocesses is a good piece of documentation on this.

[geoagr2003]

Thanks for the advises @Ciaran_Evans I’ll come back on this page with a feedback after I’ll make a try with the proposed solutions

[Ciaran_Evans]

No problem, hopefully it helps!

[msi-sslee]

Hey. So I understand this has been 4 years ago but has this ever been resolved?
Does product.dispose() also clear the product from the cache as well?

[roviesent]

Sorry to keep bringing this one up, is there any other suggestions of ways to clear this cache? It seems each method in the posts above is not effective.

[diana_harosa]

@dolaf could you have a look? Thanks

[dolaf]

Hi, from the latest post of @Ciaran_Evans I understand that his workaround was successful at that time. However, in the meantime snappy has been significantly revised for the new SNAP 10, so we need to check the issue in the latest SNAP 10 (release candidate) environment. @roviesent: could you describe your particular use case and problem in more detail so that we can try to reproduce? Thanks!