java.lang.NullPointerException when using Calibration Operator on Beam/DIMAP file

[kedziorm]

I created subset of my data and saved it in Beam/DIMAP format.
I am able to open this product in SNAP Desktop, display it’s bands as well as calculate histograms for them, so I guess that file is not corrupted.

Unfortunately it seems that regardless of using Calibration Operator in SNAP Desktop nor calling it from Python, I receive an error message.

I’m attaching my test data as well as some print screens to prove that file is usable and provide error messages:
https://drive.google.com/folderview?id=0B-YdC3R_jRUbQ0NKWUZOOXlEdW8&usp=sharing

[marpet]

It seems that in the metadata the ‘calibration’ element is missing.

java.lang.NullPointerException
   at org.esa.s1tbx.calibration.gpf.calibrators.Sentinel1Calibrator.getCalibrationVectors(Sentinel1Calibrator.java:204)
   at org.esa.s1tbx.calibration.gpf.calibrators.Sentinel1Calibrator.getVectors(Sentinel1Calibrator.java:189)
   at org.esa.s1tbx.calibration.gpf.calibrators.Sentinel1Calibrator.initialize(Sentinel1Calibrator.java:151)
Caused: org.esa.snap.core.gpf.OperatorException
   at org.esa.s1tbx.calibration.gpf.calibrators.Sentinel1Calibrator.initialize(Sentinel1Calibrator.java:160)
   at org.esa.s1tbx.calibration.gpf.CalibrationOp.initialize(CalibrationOp.java:146)
Caused: org.esa.snap.core.gpf.OperatorException: org.esa.snap.core.gpf.OperatorException due to    java.lang.NullPointerException
   at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:421)
   at org.esa.s1tbx.calibration.gpf.CalibrationOp.initialize(CalibrationOp.java:159)
   at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:485)
   at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:272)
...

I don’t know where it originally should come from. Maybe it got lost during subset?

I also think that the cause should be passed to the new OperatorException created in OperatorUtils.catchOperatorException(). Otherwise the original stack trace gets lost.

@lveci can you have a look?

[mengdahl]

As a general principle orbits and calibration should be taken care of first (even before subset), so doing the operations in different order could fix the issue.

[lveci]

The original product’s metadata is missing. The calibration needs these to get the calibration vector.
Go back to the original product and retrace how the metadata got lost. You might have selected in the creation of the subset to remove the original product metadata. I recall there was a bug with this in a very early version of SNAP. What version was the subset created with?
The current subset operation keeps the metadata by default and so the calibration should work with it.

[kedziorm]

What version was the subset created with?

I created subset using snappy package.
Typing following commands:

python
import pkg_resources
pkg_resources.get_distribution("snappy").version

returns:
'2.0.0'
so I guess I have snappy in version 2.0.0 (should I upgrade it and how?)

I created subset using my scripts on Github, by calling:

getSubset(SentinelFile)

I used following code:

import os

# To avoid RuntimeError: java.lang.OutOfMemoryError: Java heap space
print(("Current _JAVA_OPTIONS: '" + os.environ.get('_JAVA_OPTIONS', 'Not Set')))
print("will be changed to '-Xmx4096m' to avoid OutOfMemoryError")
os.environ["_JAVA_OPTIONS"] = "-Xmx4096m"
os.system('export _JAVA_OPTIONS=-Xmx4096m')

##### Global parameters:
# output files format:
OutputType = [".dim", "BEAM-DIMAP"]
# Area - Polygon should describe part of the Eastern Poland
wkt = "POLYGON((23.00 52.00,24.00 52.00,24.00 52.25,23.00 52.25,23.00 52))"
# prefixes added to file names:
prefixes = ["calibrated", "subset"]

import snappy
from snappy import ProductIO
#from snappy import GPF
from snappy import jpy

# Sample file used in testing:
from os.path import expanduser
home = expanduser("~")
SentinelPath = os.path.join(home, "Testy")
SentinelFile = os.path.join(SentinelPath,
"S1A_IW_GRDH_1SDV_20160512T161044_20160512T161109_011228_010FA8_C584.zip")

def newFilepath(Filepath, prefix):
    return os.path.join(os.path.dirname(Filepath),
    "_".join([prefix, os.path.basename(Filepath)[0:45]]) + OutputType[0])

def getSubset(SentinelFile):
    #Initialize:
    SubsetOp = snappy.jpy.get_type('org.esa.snap.core.gpf.common.SubsetOp')
    WKTReader = snappy.jpy.get_type('com.vividsolutions.jts.io.WKTReader')
    geom = WKTReader().read(wkt)
    op = SubsetOp()
    # read source product and set properties:
    product = ProductIO.readProduct(SentinelFile)
    op.setSourceProduct(product)
    op.setGeoRegion(geom)
    sub_product = op.getTargetProduct()
    # Ensure that file does not exists:
    newFile = newFilepath(SentinelFile, prefixes[1])
    if os.path.exists(newFile):
        print("It seems that subset of your data already exists. Bye!")
    else:
        print(("Starting writing to the file: " + newFile))
        ProductIO.writeProduct(sub_product, newFile, OutputType[1])
    product.dispose()
    sub_product.dispose()
    return newFile

[marpet]

The version returned by
pkg_resources.get_distribution("snappy").version
hasn’t been updated for a while. But I think you use the one provided with SNAP 4.0, right?

Depending on how you configured it, snappy might still use an older installation of SNAP. If it still exists. Maybe you have to redo the configuration. Or there is an other reason why the metadata is absent. Can you check if the metadata is present when you open the product in SNAP. Especially the calibration elements. And please also check what happens if you do the subsetting in SNAP.
However it would be better if the error would be more meaningful and give more information instead just the plain NPE.

[kedziorm]

@marpet I have completely fresh installation of SNAP (no previous installations, since I reinstalled whole Ubuntu OS from scratch).

As I checked it is true that subset does not contain calibration metadata.

As suggested by @mengdahl changing order of reprocessing (calibration first, then subsettings) help to avoid mentioned issue:

getSubset(getSigma(SentinelFile))

[Lukas]

When you say that subsetting should be done after calibration, does that mean I can do it before terrain flattening and terrain correction?

Right now my pre-processing chain is: orbit-TNR-speckle-calibration-TerrainFlattening-TerrainCorrection

[lveci]

Don’t apply anything that filters the data like speckle filter before calibration.

[Lukas]

This bug isn’t resolved in snappy. When accessing the subset command (org.esa.snap.core.gpf.common.SubsetOp) from snappy, the original product metadata is definitely not copied automatically. Adding the line op.setCopyMetadata(True) (where op is the subset operation object) solves the problem.

[marpet]

That the metadata is not copied by the subset operator is not a bug, at least it is not a problem of snappy. As snappy simply provides the SNAP API for the usage with Python.
The Subset operators copies the metadata only if the parameter is set. This is the intended behaviour, since ~10 years. This behaviour might be questionable but it’s not a bug.

[Lukas]

Thanks for the crucial clarification!