I bumped my VM up to 8GB and also bumped up the java_max_mem property in snappy.ini to 8G
I’m processing some S1 imagery and subsetting it to POLYGON ((-4.136685132980347 50.36611236042063, -4.12860631942749 50.36611236042063, -4.12860631942749 50.37036895441764, -4.136685132980347 50.37036895441764, -4.136685132980347 50.36611236042063)) so its very small in terms of a S1 image.
Running the process once I get python hogging 16% of the VMs memory, making several calls it always idled at more memory:
I’m assuming this is to do with the Java side because memory just isn’t being freed. I’m removing all files in /tmp every time I run the process too so. Is there anyway to stop this continuous buildup of memory till eventual crashing?
Further to this, I’ve noticed posts mentioning forcing garbage collection via something such as jpy.get_type('java.lang.System').gc() and this doesn’t seem to do anything. It actually seems to increase the memory usage…
Snappy is known to have memory issues, especially with clearing the memory after running a script. What might be worth a try is running your python script from Command Window, instead of Spyder (i particularly experienced problems with Spyder).
The script is being running through a Flask application http://flask.pocoo.org so it’s not a issue with Spyder.
I’m hosting this as an api to call so having it effectively bomb out after a few calls isn’t handy at all, so far my only option is to write a cron job to kill the process and restart it, but that’s basically making the api useless as it could cut out when it’s running a job.
Frustrating that trying to force garbage collection doesn’t seem to help.
No, I have no clue. Sorry. I also don’t see meaningful improvements coming very soon.
I still think that the major cause is the S1 data, respectively the S1 reader. But I’m not sure.
Garbage collection did not help me either. Maybe something related to this? I had a similar memory issues because I was invoking jpy.get_type() inside a loop. It is better to move out of the loop. It solved the issue for me.
Cheers
jose
I’ve created one now. (SNAP-960). For me it is not yet clear where the memory issues comes from. Maybe it is within SNAP, maybe in jpy or in snappy. This needs to be investigated.
However, I can tell that the further development of SNAP and the Toolboxes will start again soon. And then we will tackle such memory and performance issues. Finally…
But till the fixes will be released it still might take a few month.
Thanks @marpet for raising the issue! Hopefully the workaround can be useful for people at the moment, I know here at work our Remote Sensing folk have hit the issue too.
@harmel I’d recommend giving the workaround ago, it’s by no means pretty but it allows for bulk processing. Give me a shout if you need a hand!
Thanks a lot @Ciaran_Evans .
I noticed that image data are loaded by chunk of 512 pixel rows. So for me the work-around is to run the process for each 512-row chunk independently.
Cheers,
Tristan
Hi Marco, just providing some diagnostic info that I hope can help you solve the issue. In my experience, the problem occurs with GPF.createProduct(). When using ProductIO.readProduct and ProductIO.writeProduct the memory issues do not appear provided you use the .dispose() method on all products you read. After using product = GPF.createProduct() the product.dispose() method is ineffective - the memory is not released. So perhaps the problem has to do with using the dispose method on the products created using GPF. Hope it helps!
Dear @MarkWilliamMatthews, @marpet
Do you confirm that you can release memory with product.dispose() after product = ProductIO.readProduct(file) and product.getBand(‘bandname’).readPixels(…) ?
From my side, it is still ineffective.
For our activities, it would be miraculous to have a working dispose() function like this:
product = ProductIO.readProduct(file)
bands = product.getBandNames()
w, h = product.getSceneRasterWidth(),product.getSceneRasterHeight()
arr = np.empty((len(bands),w,h))
for i,band in enumerate(bands):
# load raster
product.getBand(band).readPixels(0,0,w,h,arr[i,...])
# release memory of the given band
product.getBand(band).dispose()