Reading downloaded data via the snappy python API


#1

My installation OSX 10.9.5, Snap 2.0 with S2TBX 2.0.3, Python 3.4

I am using the snappy python interface to read S2 data in order to generate time series of values at specific locations.

I have downloaded some data and this comes in the form of a directory ending .SAFE
(eg: S2A_OPER_PRD_MSIL1C_PDMC_20151120T075849_R002_V20151120T010117_20151120T010117.SAFE)
with a hierarchy of files underneath, though this does not include any “.dim” file.

All the python examples I have seen appear to work with “.dim” files. For example in the following code:

from snappy import ProductIO
p = ProductIO.readProduct(test_file)

I have tried this code supplying the name of the directory and of jp2 files but this simply returns a None object. As there are no “dim” files it isn’t clear to me how I can load the data under the SAFE directory.

Question: How can I do this within Python? Should I be downloading data in a different (dim) form?

In another thread, I have seen this code:

file = snappy.File(fileNameString)
readerString = ‘Sen3_SLSTRL1B_1km’
product = snappy.ProductIO.readProduct(file, readerString)

Question: Where I can get a list of accepted readerString values for S2tBX?

Thanks in advance for any help…
Olly Morgan


#2

You can not open the jp2-Files with the readProduct (as you can not open them in SNAP either, see Topic Sentinel 2 product opener). You need to open the following XML-File:

C:/
…’— S2
…’— S2A_OPER_PRD_MSIL1C_PDMC_…20151217T103953 <-- Your Directory of the S2-Dataset
…|---- AUX_DATA
…|---- DATASTRIP
…|---- GRANULE
…|---- HTML
…|---- rep_info
…|---- INSPIRE.xml
…|---- manifest.safe
…|---- S2A_OPER_BWI_MSIL1C_PDMC
20151217T103953.png
…’---- S2A_OPER_MTD_SAFL1C_PDMC
…_20151217T103953.xml <-- open this XML-file in SNAP/snappy (readProduct)

Cheers,
Andreas


#3

I use this post, because my topic is quite similar.

I run the testcode without any problems, thus I want to read an actual data set. Therefor I downloaded the S2A_MSIL1C_20170125T102141_N0204_R036_T24CWC_20170125T102143.zip and unpacked it to S2A_MSIL1C_20170125T102141_N0204_R036_T24CWC_20170125T102143.SAFE . I copied the .xml-file, which @abgbaumann mentioned above, into the snappy-directory and tried the following code:
from snappy import ProductIO
>>> product=ProductIO.readProduct("MTD_MSIL1C.xml")
But I got the following error multiple times:
SEVERE: org.esa.snap.core.dataio.ProductIO: Error attempting to read test.xml with plugin reader org.esa.s2tbx.dataio.s2.ortho.plugins.Sentinel2L1CProduct_Multi_UTM51N_ReaderPlugIn@4a9e6faf: The product ~/.snap/snap-python/MTD_MSIL1C.xml was not parsed or does not contain any granule

Thus, I thought I have to copy the whole directory with all the data.

>>> product=ProductIO.readProduct("/S2A_MSIL1C_20170125T102141_N0204_R036_T24CWC_20170125T102143.SAFE/MTD_MSIL1C.xml")

But I got the same error multiple times.
SEVERE: org.esa.snap.core.dataio.ProductIO: Error attempting to read ./S2A_MSIL1C_20170125T102141_N0204_R036_T24CWC_20170125T102143.SAFE/MTD_MSIL1C.xml with plugin reader org.esa.s2tbx.dataio.s2.ortho.plugins.Sentinel2L1CProduct_Multi_UTM60S_ReaderPlugIn@1950e8a6: The product ~/.snap/snap-python/./S2A_MSIL1C_20170125T102141_N0204_R036_T24CWC_20170125T102143.SAFE/MTD_MSIL1C.xml was not parsed or does not contain any granule

I have no idea what is wrong. Can someone tell me how to read the downloaded xml-files with snappy?
Greetings, Mara


#4

Your second approach was the right one. the whole directory is necessary.
Actually you don’t need to copy the data around. You can specify the absolut path to the xml file in python.
Have you tried to open the file in SNAP Desktop?
Maybe the download is corrupted?

@obarrilero Do you now how this can happen?


#5

Hi,

It seems that you have not copied properly the GRANULE folder. Could you please try to open the product by using the full path to the original product?


#6

Hi,

first I opened the file in SNAP Desktop and it worked.
I know I don’t need to copy the file, but I wanted to try it first this way, because I’m kind of lazy and don"t want to use the whole path. :sweat_smile:

I tried it with the whole path and used the file, where I unzipped the downloaded file, but I got a lot of error code.

>>> from snappy import ProductIO
>>> product=ProductIO.readProduct("~/01_Arbeit/Daten-Antarktis/S2A_MSIL1C_20170125T102141_N0204_R036_T24CWC_20170125T102143.SAFE/MTD_MSIL1C.xml")
java.lang.UnsatisfiedLinkError: no jhdf5 in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at ncsa.hdf.hdf5lib.H5.loadH5Lib(H5.java:339)
	at ncsa.hdf.hdf5lib.H5.<clinit>(H5.java:266)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.vito.probavbox.dataio.probav.ProbaVProductReaderPlugIn.loadClassWithNativeDependencies(ProbaVProductReaderPlugIn.java:171)
	at org.vito.probavbox.dataio.probav.ProbaVProductReaderPlugIn.loadHdf5Lib(ProbaVProductReaderPlugIn.java:106)
	at org.vito.probavbox.dataio.probav.ProbaVProductReaderPlugIn.<clinit>(ProbaVProductReaderPlugIn.java:39)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at org.esa.snap.SnapCoreActivator.loadServices(SnapCoreActivator.java:60)
	at org.esa.snap.core.dataio.ProductIOPlugInManager.<init>(ProductIOPlugInManager.java:193)
	at org.esa.snap.core.dataio.ProductIOPlugInManager$Holder.<clinit>(ProductIOPlugInManager.java:199)
	at org.esa.snap.core.dataio.ProductIOPlugInManager.getInstance(ProductIOPlugInManager.java:45)
	at org.esa.snap.core.dataio.ProductIO.getProductReaderForInput(ProductIO.java:255)
	at org.esa.snap.core.dataio.ProductIO.readProductImpl(ProductIO.java:214)
	at org.esa.snap.core.dataio.ProductIO.readProduct(ProductIO.java:186)
WARNING: org.vito.probavbox.dataio.probav.ProbaVProductReaderPlugIn: class org.vito.probavbox.dataio.probav.ProbaVProductReaderPlugIn: HDF-5 library not available: class java.lang.UnsatisfiedLinkError: ncsa.hdf.hdf5lib.H5.H5dont_atexit()I
WARNING: org.esa.snap.dataio.hdf5.Hdf5ProductWriterPlugIn: class org.esa.snap.dataio.hdf5.Hdf5ProductWriterPlugIn: HDF-5 library not available: class java.lang.NoClassDefFoundError: Could not initialize class ncsa.hdf.hdf5lib.H5
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/zmaw/sw/squeeze-x64/snap-5.0-python35/snap/modules/ext/org.esa.snap.snap-netcdf/org-slf4j/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/zmaw/u300809/.snap/system/modules/ext/org.esa.snap.snap-netcdf/org-slf4j/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32724
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist

I"m not really sure but I’m working in python3.5. I know you support only python3.4. Can this be the problem? And we use snap 5.0.


#7

The “no jhdf5 in java.library.path” is knwon. The hdf native libs can’t be loaded properly from python. As long you don’t want to open an hdf file you can ignore it.
The Python 3.5 issue is mainly on windows. On Unix it should work.


#8

Hei,

I got a similar problem with the readProduct in python, I get this error multiple times (different UTMXXX):

SEVERE: org.esa.snap.core.dataio.ProductIO: Error attempting to read testdata/MER_FRS_L1B_SUBSET.dim with plugin reader org.esa.s2tbx.dataio.s2.ortho.plugins.Sentinel2L1CProduct_Multi_UTM46S_ReaderPlugIn@2c177f9e: The product /home/ubuntu/.snap/snap-python/snappy/testdata/MER_FRS_L1B_SUBSET.dim was not parsed or does not contain any granule

I tried with my own downloaded Sentinel-2 data (.SAFE) and the testdata (.dim), always the same, except with the testdata there is some product in the variable that i opened it to, with the Sentinel2 data it stays empty.
I also tried to open the Sentinel-2 .xml file within the .SAFE, there I get:

INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32634
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.s2tbx.dataio.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist

The Sentinel-2 file should be ok, I can open them in snap on another machine without any problems.

I am working on an external virtual ubuntu machine without display only via putty, so the snappy configuration was a little workaround by configuring it on my ubuntu laptop and then changing the paths in the files in the snappy directory. (I just found out that there is no JAVA_HOME variable set, could that be influencing this?is that something that should happen during configuration?)

In case it matters I am working with Anaconda2 4.2.0 (with newer version sen2cor didnt work), python 2.7, snap 5.0.

I hope you can help :slight_smile:

-Samantha


#9

Your specifying the Path to a MERIS subset but trying to read it with the Sentinel-2 reader.
Could this be the problem?


#11

Ok, apparantly yes. And apparantly after doing the same thing for ten times, not working, its now working, both the MERIS and the Sentinel-2 file. Computer magic. Sorry for taking your time and thanks for the fast help!

One last question: could the unset JAVA_HOME variable be a problem for something?


#12

So I guess you specify the S2 format in the readProduct method, in order to read on a certain resolution or UTM zone.
If you remove the S2 format you can read other formats.
You might need to distinguish between the formats.

This is a little abstraction which simplifies things:

def readProduct(path, formatName=None):
    from snappy import (File, ProductIO)
    productFile = File(path)
    if formatName is None:
        return ProductIO.readProduct(productFile)
    else:
        reader = ProductIO.getProductReader(formatName)
        return reader.readProductNodes(productFile, None)

JAVA_HOME not set, should not be a problem.
Because SNAP brings it’s own Java.


#13

Like the first post in this thread I’ve experienced troubles reading downloaded data (.SAFE) with the snappy python API.

from snappy import ProductIO
p = ProductIO.readProduct(test_file)

Error messages looking like a long list containing all permutations of readerplugins.

SEVERE: org.esa.snap.core.dataio.ProductIO: Error attempting to read snappy/testdata/MER_FRS_L1B_SUBSET.dim with plugin reader org.esa.s2tbx.dataio.s2.ortho.plugins.Sentinel2L1CProduct_Multi_UTM52S_ReaderPlugIn@3bffddff: The product /home/ingarlokal/.snap/snap-python/snappy/testdata/MER_FRS_L1B_SUBSET.dim was not parsed or does not contain any granule

These errors were also present for different types of products.

With similar errors and several suggestions being reported in threads on the forum, it took me a while to figure out that none of these seemed to solve my issue. I finally figured it out though.

It turns out that Product().readProduct(test_file) does not accept a relative file path (!..). Switch to absolute file path and problem solved.

The error messages are quite misleading though. A file path to a non-existent file will give a file not found exception, as expected. With a relative path, snappy therefore seems to identify the correct file, but fails while indicating a parsing error as opposed to the real issue.

Btw. My setup was snap version 5 on ubuntu 16.04.

cheers