Error when performing SliceAssembly with S1 images acquired in May 2016

davosnitti · November 9, 2020, 3:28pm

Hello everybody,
I’m using SNAP 8.0.
The utility “Check for updates” has been performed also.
Now, I’m trying to assembly the following two slices:
S1A_IW_SLC__1SDV_20160504T171448_20160504T171515_011112_010BED_965C
S1A_IW_SLC__1SDV_20160504T171423_20160504T171450_011112_010BED_04A2
The two slices are available at the following links:
https://scihub.copernicus.eu/dhus/odata/v1/Products(‘a874fb02-f6ef-4f70-b7b6-f394ae6503b5’)/$value
https://scihub.copernicus.eu/dhus/odata/v1/Products(‘babcf95e-1ea8-41e8-9eb8-49c7f1fed949’)/$value

When I try to assembly them, I get the following error:

Please, anybody has any suggestion to attempt to solve this issue?
Thanks in advance
davide

ABraun · November 9, 2020, 5:54pm

does this error occur for other image pairs as well or only this one?

If so, have you checked if the data was correctly downloaded? Try to open a sub-swath of each of them and see if it can be displayed.

davosnitti · November 9, 2020, 6:27pm

Dear Andreas,
thanks a lot for your reply.

Yes, I have checked successfully the integrity of the two zip-files, by comparing the md5sum. No problem at all when trying to open any sub-swath of each of them!

Furhermore, the error occurs by processing the two products stored in the SCIHUB, as well as the corresponding products stored in the CREODIAS (unzipped, in the last case).

I’m processing a stack of image pairs, acquired on relative orbit 015 since April 2015 to date (so, more than 250 acquisition dates). The reported error does not occur sistematically for all the acquisitions but only for a number of around 30 pairs acquired between April 2015 and May 2016.
So it is not an “isolated pair”, but this error impacts on a significant number of SLC pairs. No errors instead when assembling slice pairs acquired later than May 2016.

Do you have any other suggestion?
Thanks
davide

PS: I’m working on servers with more than 32GB of RAM, and the error occurs on both Linux and Windows platforms, and by using the snap graphic interface, as well as the gpt batch script

ABraun · November 9, 2020, 6:49pm

thank you for the detailled feedback.
Doesn’t sound like a processing error then, but rather caused by the datafiles themselves. Probably only one of the two is faulty… I’m not sure if there are other S1 products around which were processed by a different facility. Could you please check if the same happens for these scenes when downloaded from ASF Data Search?

davosnitti · November 10, 2020, 7:29am

Unfortunately, the problem arises also with products downloaded from the “ASF Data Search” Facility

mengdahl · November 10, 2020, 10:00am

@lveci if this is a bug it should be fixed in the first module update.

davosnitti · November 10, 2020, 10:07am

Update:
I have downloaded two more products belonging to the same track, so now I have:
[1] S1A_IW_SLC__1SDV_20160504T171359_20160504T171426_011112_010BED_DC8E.zip
[2] S1A_IW_SLC__1SDV_20160504T171423_20160504T171450_011112_010BED_04A2.zip
[3] S1A_IW_SLC__1SDV_20160504T171448_20160504T171515_011112_010BED_965C.zip
[4] S1A_IW_SLC__1SDV_20160504T171513_20160504T171540_011112_010BED_F8B2.zip

Then, I tried to assembly subsequent pairs, with the following results:
[1]-[2]: SLICE_ASSEMBLY COMPLETED SUCCESSFULLY
[2]-[3]: SLICE_ASSEMBLY COMPLETED WITH ERROR (exactly what I reported before)
[3]-[4]: SLICE_ASSEMBLY COMPLETED SUCCESSFULLY

I’m wondering if the problem concerns the software (SNAP) or the S1 products.
Unfortunately, ther is not a verbose logfile that I can copy and paste here.
That’s all, at the moment …
Thanks
davide

davosnitti · November 12, 2020, 2:35pm

Any news or suggestions?

ericloewe · November 12, 2020, 3:40pm

The IPF was updated around that time to fix an issue which affected slice boundaries. If the manifest.safe reports or earlier, you might be running into that issue, solved in 2.71. If the image was reprocessed with a later version, this should not apply.
Version 2.71 was deployed for products after 2016-05-11 12:00:00. The timing is suspicious…

If this is indeed your problem, there’s not much you can do. The L0 data would need to be reprocessed.

davosnitti · November 13, 2020, 11:48am

Dear Eric,
thanks a lot for your reply.
I have analyzed the manifest files of all the SLC products: the IPF version of all of them is higher than 2.70, ranging from 2.71 to 3.31.
So, it seems that the problem you have mentioned does not apply in my case.

However, I have to report that the error occurs systematically for all and only the SLC slice pairs that have been focused with a different IPF version. More specifically, this happens with the IPF versions 3.10 and 2.72, respectively for the first and second slice!!

Here below I list all the slices where the error occurs, together with the corresponding IPF release:

S1A_IW_SLC__1SDV_20150404T171421_20150404T171448_005337_006C3A_FF59 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150404T171446_20150404T171513_005337_006C3A_BACC [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150416T171422_20150416T171449_005512_00709D_A62B [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150416T171447_20150416T171514_005512_00709D_755D [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150510T171423_20150510T171450_005862_0078B5_3715 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150510T171448_20150510T171511_005862_0078B5_3EC8 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150522T171424_20150522T171451_006037_007CBA_CB75 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150522T171449_20150522T171516_006037_007CBA_3050 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150603T171425_20150603T171452_006212_0081AF_4488 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150603T171450_20150603T171512_006212_0081AF_D8B9 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150615T171426_20150615T171453_006387_0086C9_F7D9 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150615T171450_20150615T171507_006387_0086C9_5DE0 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150627T171426_20150627T171453_006562_008BB6_FBF6 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150627T171451_20150627T171514_006562_008BB6_4E78 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150709T171426_20150709T171453_006737_009069_84B4 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150709T171451_20150709T171507_006737_009069_676B [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150721T171427_20150721T171454_006912_00957B_9A46 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150721T171452_20150721T171509_006912_00957B_2FFB [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150814T171428_20150814T171455_007262_009F26_B84C [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150814T171453_20150814T171510_007262_009F26_1FC0 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150826T171429_20150826T171456_007437_00A3E8_75B8 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150826T171454_20150826T171510_007437_00A3E8_6B98 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150907T171429_20150907T171456_007612_00A8B1_7AD2 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150907T171454_20150907T171511_007612_00A8B1_66B5 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20150919T171429_20150919T171456_007787_00AD57_7BB0 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20150919T171454_20150919T171511_007787_00AD57_2475 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151001T171430_20151001T171457_007962_00B212_8908 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151001T171455_20151001T171511_007962_00B212_7530 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151013T171429_20151013T171456_008137_00B6BC_D93A [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151013T171454_20151013T171512_008137_00B6BC_3212 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151025T171430_20151025T171457_008312_00BB87_5CBC [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151025T171455_20151025T171522_008312_00BB87_BAAF [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151106T171430_20151106T171457_008487_00C017_9424 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151106T171454_20151106T171521_008487_00C017_7C8B [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151118T171424_20151118T171451_008662_00C501_F517 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151118T171449_20151118T171516_008662_00C501_C721 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151130T171429_20151130T171456_008837_00C9E6_5796 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151130T171454_20151130T171521_008837_00C9E6_A2EC [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151212T171423_20151212T171450_009012_00CEC5_400B [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151212T171448_20151212T171515_009012_00CEC5_6FB1 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20151224T171428_20151224T171455_009187_00D3BB_194C [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20151224T171453_20151224T171520_009187_00D3BB_0E6F [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160117T171427_20160117T171454_009537_00DDB4_859D [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160117T171452_20160117T171519_009537_00DDB4_82DF [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160129T171421_20160129T171448_009712_00E2DB_6FCF [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160129T171446_20160129T171513_009712_00E2DB_0785 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160210T171427_20160210T171454_009887_00E7E2_90E1 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160210T171451_20160210T171518_009887_00E7E2_FEDF [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160305T171427_20160305T171454_010237_00F1FA_DEA6 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160305T171452_20160305T171519_010237_00F1FA_FF21 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160317T171422_20160317T171449_010412_00F6EF_FD1D [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160317T171446_20160317T171513_010412_00F6EF_76F7 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160329T171428_20160329T171454_010587_00FBF9_2478 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160329T171452_20160329T171519_010587_00FBF9_CEF2 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160410T171422_20160410T171449_010762_010122_A11F [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160410T171447_20160410T171514_010762_010122_849F [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160422T171423_20160422T171450_010937_010677_168F [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160422T171448_20160422T171515_010937_010677_CCA8 [IPF ver.:2.72]

S1A_IW_SLC__1SDV_20160504T171423_20160504T171450_011112_010BED_04A2 [IPF ver.:3.10]
S1A_IW_SLC__1SDV_20160504T171448_20160504T171515_011112_010BED_965C [IPF ver.:2.72]

I notice also that the error does not occur in all the cases where it happens the contrary, i.e. the first slice is focused with IPF ver. 2.72 and the second one with version 3.10.
This is the case of the following SLC pairs:

S1A_IW_SLC__1SDV_20150428T171413_20150428T171440_005687_0074BD_3B88 [IPF ver.:2.72]
S1A_IW_SLC__1SDV_20150428T171437_20150428T171510_005687_0074BD_87A1 [IPF ver.:3.10]

S1A_IW_SLC__1SDV_20150802T171417_20150802T171444_007087_009A61_5395 [IPF ver.:2.72]
S1A_IW_SLC__1SDV_20150802T171442_20150802T171509_007087_009A61_4D3D [IPF ver.:3.10]

Finally, no errors are reported when the IPF version is the same for the two slices.

I hope this can help to find a fix …
I wonder again if it is a bug of the SNAP software rather than an inconsistency that just requires L0 data to be reprocessed.
Any suggestion?
Thanks again
Davide

PS: obviously, the results do not change if I try to invert the order of the images to provide in input to the SliceAssembly operator

davosnitti · November 18, 2020, 3:59pm

Dear all, any good news ?

mengdahl · November 18, 2020, 4:13pm

I’m not sure if this is a bug or an issue of IPF versions incompatibility when trying the stich together a single data take. @nuno.miranda is this is expected behaviour in your opinion?

nuno.miranda · November 19, 2020, 7:27am

Dear Marcus and all,

It is clear that some slices of the same data-take has been reprocessed with a newer IPF version. In general, we pass the message to our PDGS colleagues to reprocess all the data-take (several slices) to avoid this situation.

Now, last week, I looked at two products reporting as giving problem. From a timing point of view it all looks consistent i.e. the burstList and swathTiming annotation doesn’t show any obvious discontinuities and similarly applies to the time sampling.

I can’t figure out any good reason for which this data can’t be stitched. I would recommend the SNAP team to perform some analysis on that. Could you please look into that direction and let me know after?

Kind Reg.
Nuno

davosnitti · January 15, 2021, 7:49am

Dear all,
any good news on this issue?
BR
don

mengdahl · January 15, 2021, 3:35pm

@lveci could you take a look? If the fix is easy perhaps it could be included in the next module update?

maawoo · October 13, 2021, 9:47am

Dear all,

I’ve stumbled upon the same or at least a very related issue with recent SLC scenes from April 2021. The IPF version of both is 3.31, so I assume there is an issue with the Slice Assembly node itself.

The error message when executing via GPT is also not very helpful unfortunately:

Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
java.lang.NullPointerException
 done.

Error: [NodeId: SliceAssembly] java.lang.NullPointerException

Edit: Got a more detailed error message after all:

Executing processing graph
java.lang.NullPointerException
 done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.3.1 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.hsqldb.persist.Logger: dataFileCache open start
org.esa.snap.core.gpf.graph.GraphException: [NodeId: SliceAssembly] java.lang.NullPointerException
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:199)
	at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:182)
	at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:166)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:85)
	at org.esa.snap.core.gpf.graph.GraphContext.<init>(GraphContext.java:58)
	at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:118)
	at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
	at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:547)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:391)
	at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
	at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
	at org.esa.snap.core.gpf.main.GPT.run(GPT.java:60)
	at org.esa.snap.core.gpf.main.GPT.main(GPT.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
	at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
	at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
	at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:84)
	at com.install4j.runtime.launcher.UnixLauncher.start(UnixLauncher.java:66)
	at install4j.org.esa.snap.runtime.Launcher_gpt.main(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
	at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:432)
	at org.esa.s1tbx.sentinel1.gpf.SliceAssemblyOp.initialize(SliceAssemblyOp.java:156)
	at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:528)
	at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:298)
	at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:385)
	at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
	... 29 more
Caused by: java.lang.NullPointerException
	at org.esa.s1tbx.sentinel1.gpf.SliceAssemblyOp.determineSliceBurstDimension(SliceAssemblyOp.java:332)
	at org.esa.s1tbx.sentinel1.gpf.SliceAssemblyOp.computeTargetWidthAndHeight(SliceAssemblyOp.java:454)
	at org.esa.s1tbx.sentinel1.gpf.SliceAssemblyOp.createTargetProduct(SliceAssemblyOp.java:522)
	at org.esa.s1tbx.sentinel1.gpf.SliceAssemblyOp.initialize(SliceAssemblyOp.java:150)
	... 33 more

Error: [NodeId: SliceAssembly] java.lang.NullPointerException