Sen-ET error in Prepare ERA5 reanalysis data

mariapina_castelli · March 17, 2021, 7:00am

Hello,

I am running the operator “Prepare ERA5 reanalysis data” from the python interface as follows:

…/bin/python3.6 ecmwf_data_preparation.py --elevation_map /mnt/CEPH_PROJECTS/FACT_ET/SNAP_SENET/T32TPS/S2A_MSIL2A_20170107/s2_elevation.dim --elevation_band elevation --ecmwf_data_file /mnt/CEPH_PROJECTS/FACT_ET/SNAP_SENET/ERA5/meteo_2017.nc --date_time_utc “2017-01-07 10:00” --time_zone 1 --prepare_temperature True --prepare_vapour_pressure True --prepare_air_pressure True --prepare_wind_speed True --prepare_clear_sky_solar_radiation True --prepare_daily_solar_irradiance True --output_file /mnt/CEPH_PROJECTS/FACT_ET/SNAP_SENET/ERA5/20170107.dim

and I get the following error:

HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) thread 140522629941056:
#000: H5T.c line 1712 in H5Tcopy(): not a datatype or dataset
major: Invalid arguments to routine
minor: Inappropriate type
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 2.4.0 set to be used by SNAP.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file /tmp/tmph1_41_9t.tif
SEVERE: org.esa.snap.core.util.SystemUtils$SnapImagingListener: JAI error occurred: ‘Problem occurs when computing a tile by the owner.’ at com.sun.media.jai.util.SunTileScheduler@233c6a98
java.lang.NullPointerException
at org.esa.s2tbx.dataio.gdal.reader.GDALTileOpImage.computeRect(GDALTileOpImage.java:64)
at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.opimage.TranslateIntOpImage.getTile(TranslateIntOpImage.java:132)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.PlanarImage.getExtendedData(PlanarImage.java:2440)
at com.sun.media.jai.opimage.MosaicOpImage.computeTile(MosaicOpImage.java:432)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.RenderedOp.getData(RenderedOp.java:2276)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.dataio.ProductIO.writeTile(ProductIO.java:602)
at org.esa.snap.core.dataio.ProductIO.lambda$writeRasterDataFully$0(ProductIO.java:578)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

SEVERE: org.esa.snap.core.util.SystemUtils$SnapImagingListener: JAI error occurred: ‘Problem occurs when computing a tile by the owner.’ at com.sun.media.jai.util.SunTileScheduler@233c6a98
java.lang.NullPointerException
at org.esa.s2tbx.dataio.gdal.reader.GDALTileOpImage.computeRect(GDALTileOpImage.java:64)
at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.opimage.TranslateIntOpImage.getTile(TranslateIntOpImage.java:132)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.PlanarImage.getExtendedData(PlanarImage.java:2440)
at com.sun.media.jai.opimage.MosaicOpImage.computeTile(MosaicOpImage.java:432)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.RenderedOp.getData(RenderedOp.java:2276)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.dataio.ProductIO.writeTile(ProductIO.java:602)
at org.esa.snap.core.dataio.ProductIO.lambda$writeRasterDataFully$0(ProductIO.java:578)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Exception in thread “pool-1-thread-1” java.lang.NullPointerException
at org.esa.s2tbx.dataio.gdal.reader.GDALTileOpImage.computeRect(GDALTileOpImage.java:64)
at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.opimage.TranslateIntOpImage.getTile(TranslateIntOpImage.java:132)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.PlanarImage.getExtendedData(PlanarImage.java:2440)
at com.sun.media.jai.opimage.MosaicOpImage.computeTile(MosaicOpImage.java:432)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at javax.media.jai.RenderedOp.getData(RenderedOp.java:2276)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.dataio.ProductIO.writeTile(ProductIO.java:602)
at org.esa.snap.core.dataio.ProductIO.lambda$writeRasterDataFully$0(ProductIO.java:578)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
air_temperature

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fcd81176705, pid=12369, tid=0x00007fcdf970c740

JRE version: OpenJDK Runtime Environment (Zulu 8.44.0.13-CA-linux64) (8.0_242-b20) (build 1.8.0_242-b20)

Java VM: OpenJDK 64-Bit Server VM (25.242-b20 mixed mode linux-amd64 compressed oops)

Problematic frame:

C [libproj.so.15+0x177705] projCppContext::getDatabaseContext()+0x45

Core dump written. Default location: /home/mcastelli@eurac.edu/.snap/auxdata/sen-et-conda-Linux64/sen-et-snap-scripts/core or core.12369

An error report file with more information is saved as:

/home/mcastelli@eurac.edu/.snap/auxdata/sen-et-conda-Linux64/sen-et-snap-scripts/hs_err_pid12369.log

If you would like to submit a bug report, please visit:

http://www.azulsystems.com/support/

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

[6] Done emacs ecmwf_data_preparation.py
Aborted (core dumped)

hs_err_pid12369.log (323.9 KB)

radosuav · March 18, 2021, 9:03am

Hi,
It looks like a problem with your meteorological HDF file:

HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) thread 140522629941056:
#000: H5T.c line 1712 in H5Tcopy(): not a datatype or dataset

Did it download properly? Maybe the AOI was too small, have a look at Error Preparing ERA 5 reanalysis data (Sen-ET plugin)

mariapina_castelli · March 18, 2021, 9:17am

Hi Rado,
this is my code line to download ERA5 data:

…/bin/python3.6 ecmwf_data_download.py --area 50.56456/3.684696/42.99109/17.16201 --start_date 20170101 --end_date 20171231 --download_path /mnt/CEPH_PROJECTS/FACT_ET/SNAP_SENET/ERA5/meteo_2017.nc --download_temperature True --download_dewpoint True --download_pressure True --download_wind_speed True --download_clear_sky_solar_radiation True --download_solar_radiation True --overwrite True

The downloaded data look ok. They cover the Alps (see attached example)

Best,
Mariapina

mariapina_castelli · March 23, 2021, 7:34am

Finally solved
I have manually configured snap8 with python 3.6 (the automatic configuration did not work), and updating the python scripts, which in the Sen-et plugin do not correspond to the last version on the GitHub repository.
Best,
Mariapina