readPixel Issue on pre-processed S3 L1B OLCI product

[smith]

Aim: Get Sentinel 3 L1B EFR data, pre-process this with the IdePix Sentinel-3 OLCI to get a cloud mask, use this mask and/or band to create a mask xarray, use this array to mask a band on another dataset of same coordinates. This is due to L1B data not having a cloud pixel flag.

python 3.10.18, esa_snappy, SNAP 11

Current Approach:
from esa_snappy import Band
print(f"\tExtracting IDEPIX_CLOUD mask…")
mask_group = subset_idepix.getMaskGroup()
idepix_cloud_mask = mask_group.get(‘IDEPIX_CLOUD’)

    if idepix_cloud_mask is not None:
        
        mask_image = jpy.cast(idepix_cloud_mask, Band)
        cloud_maskdata = np.zeros((height, width), dtype=np.int16)
        mask_image.readPixels(0, 0, width, height, cloud_mask_data)

  or by just getting the band of the idepix product (geometry subset):

quality_flags = product.getBand(‘pixel_classif_flags’), returns org.esa.snap.core.datamodel.Band[pixel_classif_flags,int16,74,98,-1,-999.0,-1,0.0,0.0,0.0] but any attempt to readPixels gives the error below too.

Tried to get more comprehensive debug logs using the advice on this previous issue:

Memory issues with Snappy

I’ve no idea why this happens. Maybe the following helps to find out what is going on. You can enable the debug mode in the snappy. ini file. Change debug to true. debug: True Now you should get some log output which might help already. If you want to get more log output you can edit the _init_.py jpy.diag.flags = jpy.diag.F_ALL

but nothing stood out in the responses other than ‘annotate_RasterDataNode_methods: Method “readValidMask”: modified parameter 4: mutable = True, return = True’ printing several times

I’ve tried several ways suggested by these similar issues:

RuntimeError: no matching Java method overloads found

I pasted your script into a file with my standard snappy preamble, replaced the filename with that of a MODIS level-2 file, and changed the polygon to wkt="POLYGON ((-78 46, -62 46, -62 30, -78 30, -78 46))". #! /usr/bin/env python3.7 # https://forum.step.esa.int/t/runtimeerror-no-matching-java-method-overloads-found/24695 import sys import os snappy_envar = 'HOME' # windows needs 'USERPROFILE' envs = os.environ if not snappy_envar in envs.keys(): raise Exception('Can’t find snappy') else: …

Reading data from boolean OLCI product Masks using snappy

Hi Marcos, The code you mentioned does not work for me now either, I think since the new version of SNAP. We have been discussing that in this thread too, but no solution so far. Thanks, Sam

such as specifying integers for the readPixels parameters, specifying floats for the cloud_mask_data both as Integer = jpy.get_type(‘java.lang.Integer’) or Float = jpy.get_type(‘java.lang.Float’) or np.int, np.float32 etc. Tried using a progress monitor as the final param but the same issue.

but keep getting the error:
no matching Java method overloads found , presumably because I may have changed int to float or data types mismatching, but even if I specify the types I may get
or ambiguous method calls instead which I think implies that because readPixels methods appear multiple times SNAP Engine API
for Band, Mask etc classes it can’t determine which one (?)

Does anyone know of a solution for this particular method or any different way to approach what I am doing to bypass this entirely (get S3 OLCI cloud mask for the same WKT geometry subset and mask another dataset of s3 processed data) ?

[Marco_EOM]

The readPixels method requires double[], float[] or int[] as type for the data. You have int16 which is short int Java. Maybe using int32 for cloud_maskdata will work. Or not using a numpy array but a python array or a list. I haven’t tried but I think this could be the cause.

[smith]

from esa_snappy import Band
mask_group = product.getMaskGroup()
idepix_cloud_mask = mask_group.get(‘IDEPIX_CLOUD’)
mask_image = jpy.cast(idepix_cloud_mask, Band)
cloud_mask_data = np.zeros(width*height, dtype=np.int32)
mask_image.readPixels(0, 0, width, height, cloud_mask_data)

or,

from esa_snappy import Band
mask_group = product.getMaskGroup()
idepix_cloud_mask = mask_group.get(‘IDEPIX_CLOUD’)
mask_image = jpy.cast(idepix_cloud_mask, Band)
cloud_mask_data = [0] * (width * height)
mask_image.readPixels(0, 0, width, height, cloud_mask_data)

give the error: ambiguous Java method call, too many matching method overloads found

same error from this alternative band method with a list:
quality_flags = product.getBand(‘pixel_classif_flags’)
width = quality_flags.getRasterWidth()
height = quality_flags.getRasterHeight()
# Use Python list instead of numpy array
cloud_mask_data = [0] * (width * height)
quality_flags.readPixels(0, 0, width, height, cloud_mask_data)

[Marco_EOM]

That’s strange. But it seems to be an older problem:

Band data types -

Hi I’m reading OLCI L3 water products (S3A_OL_2_WRR* and S3A_OL_2_WFR*) with snappy. All the bands’ data are stored as integers with a scaling factor and an offset. The call readvar.readPixels(0,0,w,h,band) reads the data as floats, thus applying the scaling factor and offset (as shown by readvar.getScalingFactor() and readvar.getScalingOffset() ) Is there a way to access the raw integers, instead of the floats? I’m trying to do something like this: read_band=p.getBand(band_name) band_t…

Maybe @dolaf can help?

[smith]

Any update on this?

[dolaf]

Hi,

sorry for the late reply, this thread got out of my focus because of other duties.

I found an older use case I worked on some time ago with a colleague who had a similar (or basically the same) issue. The problem was/is that jpy always wants to access the read_pixels(int, int, int, int, float[]) method, even if you specify an integer type for the array to read into (as you did with np.int16 for your cloud mask data. The reason remained unclear for us and thus the basic problem could not be resolved.

However, a successful workaround for us was to just read the data into a Numpy float array, and afterwards cast into an uint32. Therefore, you may try to change like this:

cloud_mask_data = np.zeros((height, width), dtype=np.int16)
mask_image.readPixels(0, 0, width, height, cloud_mask_data)

to

cloud_mask_data = np.zeros((height, width))
mask_image.readPixels(0, 0, width, height, cloud_mask_data)
cloud_mask_data = cloud_mask_data.astype(np.uint32)

We hope this suggestion helps.

Cheers,
Olaf

[smith]

Thank you for getting back to me. In the end, I did a long way round:
band = p.getBand(‘pixel_classif_flags’)
idepix_flags = band.readPixels(0, 0, w, h, zeros((h, w), float32)).astype(‘uint32’)

then getting the bits with the method described here:

Sentinel 3 IDEPIX mask in snappy

Hey @GeorgeB Different approach, but maybe try this: from numpy import zeros, where, float32, nan # Function to return the bit number from a value def get_bit(value, bit_number): """ Return value from a bit number """ return (value & (1 << bit_number)) != 0 # Read in the product p = ProductIO.readProduct(idepix.dim) # Get product dimensions h = int(p.getSceneRasterHeight()) w = int(p.getSceneRasterWidth()) # Get the flags band (this is bits) idepix_flags = p.getBand('pixel_…

then applying the mask by extracting the band data one by one:
for loop etc…
data = np.zeros(width * height, dtype=np.float32)
band.readPixels(0, 0, width, height, data)
data.shape = height, width
band_data[band_name] = data

then for loop etc…
data[combined_mask] = np.nan

very long winded but I think it achieved what I wanted in the meantime. Will probably attempt the method you gave when I get time and if it works move to that.

[viszontpadlas]

Hello everyone, sorry for posting here—I’m not very familiar with forums, but I have to turn to you because I can’t find a solution to my problem. For my thesis, I need various data from satellite images so I can calculate indices, so I downloaded SAFE files from the Copernicus Browser. In the latest version of SNAP (and I tried the previous one as well), it won’t open the XML called MTD_MSIL2A; I load it, but nothing happens. I’d like to ask for help—what should I do in this situation?

[diana_harosa]

Dear @viszontpadlas,

Please attach the messages.log file generated after reproducing the issue.

Steps to follow:

1. Launch SNAP.
2. Try to open the S2 product using the MTD_MSIL2A.xml file.
3. Go to Help → Show Log Directory.
4. Attach the messages.log file from that directory.

[viszontpadlas]

messages.log (102.2 KB)

[diana_harosa]

Dear @viszontpadlas,
The issue seems to be related to your user directory, which contains a special character (á). As a workaround, please move the SNAP working directory to another location outside your user directory.

To do this, update the C:\Program Files\esa-snap\etc\snap.conf file by adding the following option:
-J-Dsnap.userdir=<SNAP_working_dir>

Example:

[viszontpadlas]

Dear Diana_Harosa,

I’m not very experienced with this topic, so I have a further question.
Could you please advise me on how to open the interface/example you sent? I can only open it with Notepad, but it appears differently to me. I’ve attached a picture for reference.

Thank you very much for your help.

Best regards,
viszontpadlas

[diana_harosa]

Hi @viszontpadlas,

Just add -J-Dsnap.userdir=<SNAP_working_dir> after the last ALL-UNNAMED entry and before the final quotation mark:

[viszontpadlas]

Dear Diana_Harosa,

I have entered what you sent and created the folders, but it still isn’t working.
Did I make a mistake, or is the problem on my side?

[diana_harosa]

After restarting SNAP, did the .snap13 folder get created in the new location?

[viszontpadlas]

No, I created it myself.

[diana_harosa]

Could you attach a screenshot of its contents and also provide the messages.log file generated after launching SNAP?

[viszontpadlas]

messages.log (105.6 KB)

[diana_harosa]

Sorry if I wasn’t clear enough. I actually wanted a screenshot of the .snap13 folder’s contents. According to the log file, SNAP isn’t using the new working directory; it’s still using the old one. Could you check whether the changes made to the configuration file were saved?

[viszontpadlas]

Dear diana_harosa,

There must have been something wrong with my previous save, because I saved it again and now it works.
Thank you very much for all your help.
Have a wonderful day.