Snaphu Export in SNAP 8 rises IndexOutOfBoundsException

luc · March 24, 2021, 3:21pm

When exporting files for Snaphu running it from Snappy python package, the IndexOutOfBoundsException exception occurs as shown below.

The exception occurs in SNAP 8. It works in SNAP 8 GUI, but fails when called from Snappy. The same code works fine in SNAP 7, but fails in SNAP 8 I am afraid it might be caused by changes in read/write introduced in SNAP 8.0 release. Could you please check what might be the cause of this error ? I will be grateful for your help.

Code example:

from snappy import GPF, ProductIO
...
# 'product' contains interferogram and all the related data
# 'parameters' specify parameters for SnaphuExport
product_snaphu = GPF.createProduct('SnaphuExport', parameters, product)
ProductIO.writeProduct(product_snaphu, output_path, 'Snaphu')

The exception:

Exception in thread "pool-1-thread-1" java.lang.IndexOutOfBoundsException
	at java.io.RandomAccessFile.writeBytes(Native Method)
	at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
	at org.esa.s1tbx.commons.io.EnhancedRandomAccessFile.flush(EnhancedRandomAccessFile.java:397)
	at org.esa.s1tbx.commons.io.EnhancedRandomAccessFile.seek(EnhancedRandomAccessFile.java:300)
	at org.esa.s1tbx.commons.io.FileImageOutputStreamExtImpl.seek(FileImageOutputStreamExtImpl.java:143)
	at org.esa.snap.core.datamodel.ProductData.writeTo(ProductData.java:1000)
	at org.jlinda.nest.dataio.SnaphuWriter.writeBandRasterData(SnaphuWriter.java:167)
	at org.esa.snap.core.datamodel.Band.writeRasterData(Band.java:359)
	at org.esa.snap.core.dataio.ProductIO.writeTile(ProductIO.java:605)
	at org.esa.snap.core.dataio.ProductIO.lambda$writeRasterDataFully$0(ProductIO.java:578)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

ChrisWong · May 19, 2021, 4:47am

My codes is from https://forum.step.esa.int/t/snaphu-read-error-due-to-non-ascii-unreadable-file/14374/3, which is similar to @luc and when I export files for Snaphu, I get the same IndexOutOfBoundsException exception in python3.6.9 Snappy package.

The exception occurs in SNAP 8 Snappy. The same code works fine in SNAP 6.0 Snappy.

I found the exception actually occurs in the code “ProductIO.writeProduct(product_snaphu, output_path, ‘Snaphu’)”, sometimes it won’t stop even the exception occurs, but then stopped when running “snaphu -f snaphu.conf Phase_ifg_*.snaphu.img 6341”, the error message is
“Unexpected or abnormal exit of child process 13076
Abort”

Grateful for any help!@ABraun

marpet · May 19, 2021, 5:55am

I think this is not related to Python. I guess this issue happens also when exporting from SNAP Desktop, right?
So, I have moved it into the S1TBX category.

ChrisWong · May 19, 2021, 8:03am

I used SNAP 8 Desktop to SnaphuExport and SnaphuInport, it works well.
And I use the command “snaphu -f snaphu.conf Phase_ifg_*.snaphu.img 6341” to process the unwrapping part instead of the Snaphu-unwraping.

The exception acturally occers in the snaphuExport part of the snappy processing.

marpet · May 19, 2021, 8:23am

Okay, then I move it back to the python category.
But still, I think the issue is probably, I don’t know, in Snaphu or in s1tbx, maybe it only occurs under certain circumstances when called from Python.
But I can’t help here. I have no knwoledge of snaphu.

luc · May 20, 2021, 5:03pm

The SnaphuExport works in SNAP 8 Desktop, but fails when called from Snappy.

The same code in Snappy works fine in SNAP 7, but fails in SNAP 8 I am afraid it is caused by changes in read/write introduced in SNAP 8. It is quite serious issue. For large scale processing you want to avoid desktop version. Running all automatically using Snappy is very convenient. This issue makes the use of SnaphuExport outside the desktop unusable. It prevents me from upgrading to SNAP 8. Please is there any core developer familiar with Snappy who could help ?

marpet · May 21, 2021, 7:03am

@lveci or @jun_lu can you investigate this? You can reproduce this issue much easier as I could and
I assume that it is not related to snappy. Snappy didn’t really change since SNAP7 and actually it just delegates to the Java methods.

jun_lu · May 24, 2021, 5:53pm

We will look into the problem. A JIRA ticket (https://senbox.atlassian.net/browse/SITBX-865) has been created to track the problem. Thanks for reporting the issue.

luc · May 27, 2021, 2:56pm

Thank you for opening the ticket. It would be great if you can give it a high priority. This issue was probably introduced by changes to write operations created in SNAP 8.0 release. It is a serious issue. It is not possible to run headless (non-gui) interferometric pipelines (which includes export for phase unwrapping). It works in SNAP 7, but SNAP 7 will not be compatible with newest versions of Ubuntu due to its dependency on libgfortran3. Thank you for taking care about this issue.

Pires · October 6, 2021, 11:58am

Did anybody found a solution for this? I am running into the same problem with Snap 8.0 with snappy.

Looked at the JIRA ticket, was no update.

andrew.poley · January 17, 2022, 9:55pm

Hello @luc, were you able to find a solution to this issue? I’m having the same problem. SNAPHU export works fine from GUI but not snappy.

Thank you,
Andrew

andrew.poley · January 18, 2022, 2:16pm

Hello, has there been any updates on this issue? Thank you,
Andrew

luc · January 19, 2022, 5:12pm

Hi Andrew, unfortunately the problem still persists. SnaphuExport fails in SNAP 8 when you are running it from Python. I have to perform this part of my processing pipeline in a docker where I still keep SNAP 7 and compatible ubuntu. Other processing steps can use SNAP 8, but this part only works in SNAP 7 If you will find a better workaround how to make it work with SNAP 8, please let me know.

andrew.poley · March 16, 2022, 7:20pm

I tried the same with just running snaphuExport in SNAP 7 and that worked. I tried running the entire process in SNAP 7 but unfortunately SNAP 7 automatic download of orbit files is broken due to the shift from ESA. This combo of SNAP 7 and 8 is all I can get to work. Thanks for the tip!

mengdahl · March 17, 2022, 10:03am

@jun_lu This issue needs attention as it’s blocking people from upgrading to SNAP 8/9.

Shaila · March 21, 2022, 5:06pm

Any update about this topic?

domteo · January 5, 2023, 12:56pm

@jun_lu The same problem with SnaphuExport is in SNAP 9. Files exported using the desktop version of SNAP 9 can be unwrapped but files exported using snappy do not and snaphu ends up at different times with this message in the console:


Unexpected or abnormal exit of child process 768254

Abort

Parent received signal 15

Killing children and exiting

Navre · March 1, 2023, 9:18am

I’m using SNAP 9, and with SNAP desktop the SnaphuExport operator works, while if using snappy I get the same error of @luc . Can someone please solve this very important issue? It’s crucial for me to build an automatic code for processing. Thanks in advance

Exception in thread “pool-2-thread-12” java.lang.IndexOutOfBoundsException
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.esa.s1tbx.commons.io.EnhancedRandomAccessFile.flush(EnhancedRandomAccessFile.java:397)
at org.esa.s1tbx.commons.io.EnhancedRandomAccessFile.seek(EnhancedRandomAccessFile.java:300)
at org.esa.s1tbx.commons.io.FileImageOutputStreamExtImpl.seek(FileImageOutputStreamExtImpl.java:143)
at org.esa.snap.core.datamodel.ProductData.writeTo(ProductData.java:1000)
at org.jlinda.nest.dataio.SnaphuWriter.writeBandRasterData(SnaphuWriter.java:167)
at org.esa.snap.core.datamodel.Band.writeRasterData(Band.java:359)
at org.esa.snap.core.dataio.ProductIO.writeTile(ProductIO.java:627)
at org.esa.snap.core.dataio.ProductIO.lambda$writeRasterDataFully$0(ProductIO.java:595)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

domteo · March 9, 2023, 2:07pm

For the time when the problem is not solved I use gpt running the graph execution

os.system(gpt export_graf.xml -Pinput1=exported.dim -Poutput1=export_path)

Koichi · June 10, 2023, 7:18am

Hello, everyone. I recently started to use SNAP 9 on Windows and am trying to do “SnaphuExport”. The issue I am facing is that “SnaphuExport” function does not give me any data as output. My code is below.

parameters = snappy.HashMap()
parameters.put("targetFolder", "../snap-python/sentinel1_SLC_processing/SNAPHU_")
parameters.put("statCostMode", "DEFO")
parameters.put("initMethod", "MCF")
parameters.put("numberofTileRows", 1)
parameters.put("numberofTileCols", 1)
parameters.put("numberOfProcessors", 4)
parameters.put("rowOverlap", 0)
parameters.put("colOverlap", 0)
parameters.put("tileCostThreshold", 0)
snaphuExport = snappy.GPF.createProduct("SnaphuExport", parameters, phaseFiltering)

I have been through some posts on forum and noticed that there was a problem in “SnaphuExport” in python-snappy of SNAP 8. Is the problem still happening in SNAP 9? Or Does anyone successfully use “SnaphuExport”? If so, I am very glad if you kindly support me.

And I saw SNAP 10 will be released in the end of this month, so this problem will be fixed.