[snap 8.0][L1CSYN 3.0] writting netcdf error while running S3 L1C SYN tool with MISR file

yoannD · April 23, 2024, 9:08am

Hello,
I am testing the S3 L1C toolbox to generate SYN_L1C products from OL1 and SL1 files using the associated MISR files.
It has worked with one of my test case, but I get the following error with my second case.

java.io.IOException: NetCDF: Not a valid ID
at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:3051)
at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2996)
at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:1068)
at org.esa.snap.dataio.netcdf.nc.N4Variable$NetCDF4ChunkWriter.writeChunk(N4Variable.java:244)
Caused: java.io.IOException
at org.esa.snap.dataio.netcdf.nc.N4Variable$NetCDF4ChunkWriter.writeChunk(N4Variable.java:246)
at org.esa.snap.dataio.netcdf.nc.ChunkWriter.write(ChunkWriter.java:82)
at org.esa.snap.dataio.netcdf.nc.N4Variable.write(N4Variable.java:206)
at org.esa.snap.dataio.netcdf.DefaultNetCdfWriter.writeBandRasterData(DefaultNetCdfWriter.java:108)
at org.esa.snap.core.gpf.common.WriteOp.writeTileRow(WriteOp.java:430)
at org.esa.snap.core.gpf.common.WriteOp.computeTile(WriteOp.java:341)
Caused: org.esa.snap.core.gpf.OperatorException: Not able to write product file: ‘/mount/opt-mpc/s3mpc/work/SNAP_L1C_SYN_TOOL/output/with_MISR/S3B_SY_1_SYN____20240226T072532_20240226T072832_20240423T080129_0179_090_106_2520_LN2_O_NT____.nc’
at org.esa.snap.core.gpf.common.WriteOp.computeTile(WriteOp.java:375)
at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
[catch] at javax.media.jai.SourcelessOpImage.computeTile(Unknown Source)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
at javax.media.jai.OpImage.getTile(Unknown Source)
at com.sun.media.jai.util.RequestJob.compute(Unknown Source)
at com.sun.media.jai.util.WorkerThread.run(Unknown Source)

I cannot figure out what is going on. would someone be able to suggest a way to overcome this issue?

my conf file looks like this:
default_options=“–branding snap --locale en_GB -J-XX:+AggressiveOpts -J-Xverify:none -J-Xms8G -J-Xmx32G -J-Dnetbeans.mainclass=org.esa.snap.main.Main -J-Dsun.java2d.noddraw=true -J-Dsun.awt.pixfmt=true -J-Dsun.java2d.dpiaware=false”

My machine has 16 cores and 64GB of RAM.
while processing up to 14 to 15 cores are used at the same time by snap using between 90 and 100% of each cpu.
Is this supposed to happen? it seems a bit overkill. The MISR file is 4.1GB, the OL1 is 797 MB and the SL1 is 427MB;
thank you in advance for your suggestions,
best regards
Yoann

yoannD · April 23, 2024, 3:17pm

hello again,
after changing a few things in the configuration parmeters such as this:
the cach size is now 16384, tile size 512, numbre of thread 10 instead of 12

i get the following errors:
java.lang.OutOfMemoryError: GC overhead limit exceeded

java.io.IOException: NetCDF: Not a valid ID
at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:3051)
at ucar.nc2.jni.netcdf.Nc4Iosp.writeData(Nc4Iosp.java:2996)
at ucar.nc2.NetcdfFileWriter.write(NetcdfFileWriter.java:1068)
at org.esa.snap.dataio.netcdf.nc.N4Variable$NetCDF4ChunkWriter.writeChunk(N4Variable.java:244)
Caused: java.io.IOException
at org.esa.snap.dataio.netcdf.nc.N4Variable$NetCDF4ChunkWriter.writeChunk(N4Variable.java:246)
at org.esa.snap.dataio.netcdf.nc.ChunkWriter.write(ChunkWriter.java:82)
at org.esa.snap.dataio.netcdf.nc.N4Variable.write(N4Variable.java:206)
at org.esa.snap.dataio.netcdf.DefaultNetCdfWriter.writeBandRasterData(DefaultNetCdfWriter.java:108)
at org.esa.snap.core.gpf.common.WriteOp.writeTileRow(WriteOp.java:430)
at org.esa.snap.core.gpf.common.WriteOp.computeTile(WriteOp.java:341)
Caused: org.esa.snap.core.gpf.OperatorException: Not able to write product file: ‘/mount/opt-mpc/s3mpc/work/SNAP_L1C_SYN_TOOL/output/with_MISR/S3B_SY_1_SYN____20240226T072532_20240226T072832_20240423T142935_0179_090_106_2520_LN2_O_NT____.nc’
at org.esa.snap.core.gpf.common.WriteOp.computeTile(WriteOp.java:375)
at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
[catch] at javax.media.jai.SourcelessOpImage.computeTile(Unknown Source)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
at javax.media.jai.OpImage.getTile(Unknown Source)
at com.sun.media.jai.util.RequestJob.compute(Unknown Source)
at com.sun.media.jai.util.WorkerThread.run(Unknown Source)

java.lang.NullPointerException
at org.esa.snap.dataio.netcdf.DefaultNetCdfWriter.getVariable(DefaultNetCdfWriter.java:155)
at org.esa.snap.dataio.netcdf.DefaultNetCdfWriter.writeBandRasterData(DefaultNetCdfWriter.java:107)
at org.esa.snap.core.gpf.common.WriteOp.writeTileRow(WriteOp.java:430)
at org.esa.snap.core.gpf.common.WriteOp.computeTile(WriteOp.java:341)
Caused: org.esa.snap.core.gpf.OperatorException: Not able to write product file: ‘/mount/opt-mpc/s3mpc/work/SNAP_L1C_SYN_TOOL/output/with_MISR/S3B_SY_1_SYN____20240226T072532_20240226T072832_20240423T142935_0179_090_106_2520_LN2_O_NT____.nc’
at org.esa.snap.core.gpf.common.WriteOp.computeTile(WriteOp.java:375)
at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
[catch] at javax.media.jai.SourcelessOpImage.computeTile(Unknown Source)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
at javax.media.jai.OpImage.getTile(Unknown Source)
at com.sun.media.jai.util.RequestJob.compute(Unknown Source)
at com.sun.media.jai.util.WorkerThread.run(Unknown Source)

I will keep tweaking but it looks like a memory allocation problem. If anyone has exeprience with this, any insight will probably help. thanks in advance,
Yoann

Marco_EOM · April 25, 2024, 5:59am

The main problem seems to be that the data cannot be written.
Have you checked this location. Do you have write-access and is there still enough disk space?

yoannD · April 25, 2024, 2:16pm

Dear Marco, thank you for your answer.
I did check indeed, no issue of permission or space. It was indeed a memory issue.
I finally got it to work, and here is the setup required on my system (this may save time to someone esle one day).

default_options=“–branding snap --locale en_GB -J-XX:+AggressiveOpts -J-Xverify:none -J-Xms8G -J-Xmx42G -J-Dnetbeans.mainclass=org.esa.snap.main.Main -J-Dsun.java2d.noddraw=true -J-Dsun.awt.nopixfmt=true -J-Dsun.java2d.dpiaware=false”

Cach size 24576 MB
Tile size 512
num threads 10

So far this has been used with a few products and all test were succesful.
Yoann