Snappy error - ProductIO.writeProduct

juanesburgo · December 28, 2015, 4:22pm

Hi,

I’m trying to implement a simple split of a Sentinel-1 product using snappy.
Here you can see my script:

import sys

sys.path.append('C:\\Users\\juanesburgo\\Desktop\\temp\\snap_python')

import snappy
from snappy import ProductIO
from snappy import jpy
from snappy import GPF

ProductIOPlugInManager = snappy.jpy.get_type('org.esa.snap.core.dataio.ProductIOPlugInManager')
Logger = jpy.get_type('java.util.logging.Logger')
Level = jpy.get_type('java.util.logging.Level')
Arrays = jpy.get_type('java.util.Arrays')
File = jpy.get_type('java.io.File')
String = jpy.get_type('java.lang.String')
HashMap = jpy.get_type('java.util.HashMap')

Logger.getLogger('').setLevel(Level.OFF)
snappy.SystemUtils.LOG.setLevel(Level.OFF)

reader_spi_it = ProductIOPlugInManager.getInstance().getAllReaderPlugIns()
writer_spi_it = ProductIOPlugInManager.getInstance().getAllWriterPlugIns()
GPF.getDefaultInstance().getOperatorSpiRegistry().loadOperatorSpis()

# file paths
ifile = 'C:\\Users\\juanesburgo\\Desktop\\temp\\dados\\S1A_IW_SLC__1SDV_20150903T050105_20150903T050132_007546_00A6E6_526A\\manifest.safe'
ofile = 'C:\\Users\\juanesburgo\\Desktop\\temp\\dados\\res\\snappy_sliped_output.dim'

# read
source = ProductIO.readProduct(ifile)

name = source.getName()
w = source.getSceneRasterWidth()
h = source.getSceneRasterHeight()
print( w, "x", h, "pixels")

# split
parameters = HashMap()
parameters.put('subswath', 'IW1')
parameters.put('selectedPolarisations', 'VV')
sliped_product = GPF.createProduct('TOPSAR-Split', parameters, source)

w = sliped_product.getSceneRasterWidth()
h = sliped_product.getSceneRasterHeight()
print( w, "x", h, "pixels")

# write
ProductIO.writeProduct(sliped_product, ofile, "BEAM-DIMAP")

When I run it I get this output:

74696 x 13716 pixels
22719 x 13563 pixels
Traceback (most recent call last):
  File "C:/Users/juanesburgo/Desktop/temp/snap_python/snappy/examples/MeusScripts/tentativa4.py", line 53, in <module>
    ProductIO.writeProduct(sliped_product, ofile, "BEAM-DIMAP")
RuntimeError: java.lang.OutOfMemoryError: Java heap space

As you can see the split is done but I can’t write the output.

Any sugestion?

Thanks.

marpet · December 29, 2015, 9:48am

I’m not sure, I’m not very familiar with snappy, but I think you have to set a value in the jpyconfig.py script in the snappy folder.
Try setting
jvm_maxmem = None
to
jvm_maxmem = '6G'

Another place to change the memory usage might be in snappy.ini in the snappy folder.
Change the line
# java_max_mem: 4G
to e.g.
java_max_mem: 6G

juanesburgo · December 30, 2015, 11:17am

Marpet, thank you.

Shahbaz · December 19, 2017, 9:15am

I am still unable to tackle with the error or memory exceeded. is there any suggestion please?

Shahbaz · December 21, 2017, 7:25am

Thanks Marpet…i got it i was on mistake

himu13 · October 29, 2021, 9:59am

Hello @marpet hope you are doing well,
I have one challenge in sentinel1 preprocessing using snappy python,
I am used one tool “Apply-Orbit-File” using manual process with SNAP Desktop as well as with the help of snappy python everything is working fine,

when I am trying to compare both output (manual and python) in Qgis software the pixel value does not match at all. I just need the output which is match with snappy python output.

can you please let me know where i have to focus on this,

The source code I have given below for your reference ;

def ApplyOrbitFuncton(S1_read, out_path):
    params = snappy.HashMap()
    params.put('orbitType',"Sentinel Precise (Auto Download)")
    params.put('polyDegree',3)
    params.put('continueOnFail', False)
    apply_orbit = snappy.GPF.createProduct('Apply-Orbit-File',params,subset)
    
    nme = "Orbit_1259"
    out = os.path.join(out_path,nme)

    snappy.ProductIO.writeProduct(apply_orbit,out, 'GeoTIFF-BigTIFF')
    print("2 Apply orbit succesfully done")
    return apply_orbit

I used all the parameters as it is used in manual process in SNAP

marpet · October 29, 2021, 10:19am

You are saying the values do not match at all.
Have you also stored the data to GeoTIFF-BigTIFF when using the Desktop and have you reload it?
It could be that the scaling factor of the bands, if they are scaled, makes the difference.
In GeoTiff only raw values are stored. Maybe you can also work with NetCDF4-CF or store the data to BEAM-DIMAP?

himu13 · October 29, 2021, 12:31pm

Sir it was fix that i have to create output same as SNAP Desktop makes,

I need to make changes in code only not in manual process

my main aim is to make output same as manual process

I have changed the output from GeoTIFF-BigTIFF to GeoTiff

but there are no changes detected.

himu13 · November 1, 2021, 5:36am

@marpet @juanesburgo
Hello everyone, Hope you all are doing well,

please let me suggest the solution for my problem that I have mention.
I have made one mistake at the first post at that time I said “I just need the output which is match with snappy python output.” but it was my mistake ,

actually I need the output which will be the same as per SNAP Desktop output.

marpet · November 1, 2021, 5:57am

@lveci Do you have an idea why the results are different in both scenarios?

himu13 · November 17, 2021, 7:31am

Problem resolved in 4steps, but same problem occur in 5th step in terrain correction;

apply orbit

thermal noise

calibration

speckle filter

terrain correction

linear to formdb

write

def terrain_correction(Speckle,out_path):
    params = snappy.HashMap()
    params.put('sourceBands','Sigma0_VH,Sigma0_VV')
    params.put('demName', 'SRTM 3Sec')
    params.put('demResamplingMethod','BILINEAR_INTERPOLATION')
    params.put('imgResamplingMethod','BILINEAR_INTERPOLATION')
    params.put('pixelSpacingInMeter', 10.0)
    params.put('pixelSpacingInDegree',0.00008983152841195215)
    params.put('mapProjection','WGS84(DD)')
    params.put('saveSelectedSourceBand',True)
    
    terrain = snappy.GPF.createProduct('Terrain-Correction', params, Speckle)
    
    nme = "Orbit_Thermal_Cali_Speckle_Terrain_test1"
    out = os.path.join(out_path,nme)
    Config = snappy.jpy.get_type('org.esa.snap.runtime.Config')
    Config.instance().preferences().put('snap.dataio.bigtiff.compression.type', 'LZW')
#    Config.instance().preferences().put('snap.dataio.bigtiff.compression.quality', '0.75')
    snappy.ProductIO.writeProduct(terrain,out, 'GeoTIFF-BigTIFF')
    print("6 terrain succesfully done")    
    return terrain

there is minor changes detected in pixel values of manual output and snappy output
@marpet @lveci can you please let me know if you have any idea ?

himu13 · April 1, 2022, 6:01am

Hello @marpet @lveci hope you are doing well.

I wanted to know how can I write image in chunk with the help of snappy.ProductIO.writeProduct ?
because full image occupy whole RAM so I want to do with chunk with multiprocessing.

gnwiii · April 2, 2022, 11:30am

This looks like a new topic, so you should start a new thread with a title chosen so others with relevant experience will respond.

Snappy may not be the most memory-efficient way to process large images. Java can do some types of image processing in smaller “tiles”. Do you need to manipulate the image using Python code? If not, you may be able to modify your existing Python code to generate an xml graph and then use Python subprocess.run() to run gpt. The snapista conda package uses this approach and is available for linux but needs a VM on Windows, so may not be practical on a system with limited RAM.