ESD issue with GPT

AriJeannin · December 9, 2020, 2:39pm

Actually not… I just managed to use subset, and it was working with interface only.
(Here I’am with other datas)

ABraun · December 9, 2020, 2:46pm

can’t think of a reason why this happened.

In case you inserted the subset operator into the graph - it could be worth a try to re-create the graph from scratch and only connect to the write operator once all steps are included. Sometimes editing existing graphs caused problems in the past.

AriJeannin · December 10, 2020, 8:52am

I did, took all bursts and it took 680 minutes (vs 5min with 3bursts). Works without any problem with interface.
Then used the same graph launched with GPT, same error as previous, but much more “Arrays must have the same length.” messages (see below)

I just guess that there is something different between interface and gpt, can’t think what neither, maybe new version issue… could be good to report to snap’s devs?

Anyway, thanks for your time!

Executing processing graph
...10%...20%...31%..-- org.jblas INFO Starting temp DLL cleanup task.
-- org.jblas INFO Deleted 4 unused temp DLL libraries from C:\Users\ajeannin\AppData\Local\Temp
SEVERE: com.sun.xml.bind.v2.util.XmlFactory: null
org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
        at org.apache.xerces.parsers.AbstractSAXParser.setFeature(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl.setFeatures(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParserImpl(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserFactoryImpl.setFeature(Unknown Source)
        at com.sun.xml.bind.v2.util.XmlFactory.createParserFactory(XmlFactory.java:106)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.getXMLReader(UnmarshallerImpl.java:124)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:214)
        at it.geosolutions.imageioimpl.plugins.tiff.gdal.GDALMetadataParser.parse(GDALMetadataParser.java:60)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.initializeFromMetadata(TIFFImageReader.java:1337)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.seekToImage(TIFFImageReader.java:865)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.seekToImage(TIFFImageReader.java:822)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.getWidth(TIFFImageReader.java:1080)
        at org.esa.snap.dataio.geotiff.GeoTiffImageReader.getImageWidth(GeoTiffImageReader.java:160)
        at org.esa.snap.dataio.geotiff.GeoTiffProductReader.readProduct(GeoTiffProductReader.java:230)
        at org.esa.snap.dataio.geotiff.GeoTiffProductReader.readProductNodesImpl(GeoTiffProductReader.java:136)
        at org.esa.snap.core.dataio.AbstractProductReader.readProductNodes(AbstractProductReader.java:178)
        at org.esa.snap.engine_utilities.gpf.CommonReaders.read(CommonReaders.java:80)
        at org.esa.snap.engine_utilities.gpf.CommonReaders.readCommonProductReader(CommonReaders.java:59)
        at org.esa.snap.engine_utilities.gpf.CommonReaders.readProduct(CommonReaders.java:33)
        at org.esa.snap.dem.dataio.FileElevationModel.init(FileElevationModel.java:53)
        at org.esa.snap.dem.dataio.FileElevationModel.<init>(FileElevationModel.java:49)
        at org.esa.s1tbx.sentinel1.gpf.BackGeocodingOp.getElevationModel(BackGeocodingOp.java:536)
        at org.esa.s1tbx.sentinel1.gpf.BackGeocodingOp.computeTileStack(BackGeocodingOp.java:486)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.getComplexDoubleMatrix(SpectralDiversityOp.java:992)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.getFineOffsets(SpectralDiversityOp.java:968)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzRgOffsets(SpectralDiversityOp.java:958)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.access$200(SpectralDiversityOp.java:110)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp$2.process(SpectralDiversityOp.java:1437)
        at org.esa.snap.core.util.ThreadRunnable.run(ThreadRunnable.java:23)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
Waiting thread received a null tile.
90% done.
org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.core.gpf.graph.GraphProcessor$GPFImagingListener.errorOccurred(GraphProcessor.java:379)
        at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(Unknown Source)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at javax.media.jai.PlanarImage.getExtendedData(Unknown Source)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:447)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:479)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computePartialTile(InterferogramOp.java:1365)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computeTileStackForTOPSARProduct(InterferogramOp.java:1290)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computeTileStack(InterferogramOp.java:863)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.cobbleFloat(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
        at org.esa.s1tbx.sentinel1.gpf.TOPSARDeburstOp.computeTileInOneSwathFloat(TOPSARDeburstOp.java:904)
        at org.esa.s1tbx.sentinel1.gpf.TOPSARDeburstOp.computeTileStack(TOPSARDeburstOp.java:811)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.cobbleFloat(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.datamodel.Band.readRasterData(Band.java:309)
        at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataRegion(ProductSubsetBuilder.java:393)
        at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataImpl(ProductSubsetBuilder.java:330)
        at org.esa.snap.core.dataio.AbstractProductReader.readBandRasterData(AbstractProductReader.java:277)
        at org.esa.snap.core.gpf.common.SubsetOp.computeTile(SubsetOp.java:297)
        at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
        at javax.media.jai.SourcelessOpImage.computeTile(Unknown Source)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:116)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at com.sun.media.jai.util.RequestJob.compute(Unknown Source)
        at com.sun.media.jai.util.WorkerThread.run(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.computeTileStack(SpectralDiversityOp.java:827)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        ... 48 more
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzimuthOffset(SpectralDiversityOp.java:1243)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.computeTileStack(SpectralDiversityOp.java:809)
        ... 50 more
Caused by: java.lang.NullPointerException
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzimuthOffset(SpectralDiversityOp.java:1117)
        ... 51 more

Error: java.lang.NullPointerException

ABraun · December 10, 2020, 10:04am

@estebanaguilera can you please check if you find a bug here?

s_andersson · January 11, 2021, 11:22am

Any update on this? I get the same error message even with a basic graph.

estebanaguilera · January 11, 2021, 6:54pm

Would you mind sharing the graph file (xml) and the name of the input products? Feel free to anonymize the paths. I can then try to reproduce/troubleshoot.

AriJeannin · January 12, 2021, 9:07am

Hi, of course, please find here an example that is working on SNAP interface but not through GPT.

inSAR_chainv0.xml (6.7 KB)

Thanks for your time!

estebanaguilera · January 12, 2021, 4:28pm

Thanks for the xml file. I ran it from both the SNAP UI and the command line and both finished successfully. I executed:

$ gpt DinSAR_chainv0.xml -c 16G -q 10

Could you maybe share the contents of your gpt.vmoptions file? and the exact command you use for running GPT?

AriJeannin · January 12, 2021, 4:54pm

Oh okay good that it works! Brings me some hope.

I don’t use the gpt.vmoptions file as I’m using a custom command with a fonction reading number of cores and max memory of the computer that is launching the script.

here with a 16cores / 64gb ram:

gpt DinSAR_chainv0.xml -q 14 -J-Xms2G -J-Xmx63G -J-Dsnap.log.level=WARNING -J-Dsnap.jai.defaultTileSize=4096 -J-Dsnap.dataio.reader.tileWidth=4096 -J-Dsnap.dataio.reader.tileHeigh=4096 -J-Dsnap.jai.prefetchTiles=true -c 47G

ps: still my gpt.vmoptions file

####### Enter one VM parameter per line
####### For example, to adjust the maximum memory usage to 512 MB, uncomment the following line:
####### -Xmx512m
####### To include another file, uncomment the following line:
####### -include-options [path to other .vmoption file]
-Xmx44G

estebanaguilera · January 12, 2021, 6:32pm

Could you try running this?

$ gpt DinSAR_chainv0.xml -c 16G -q 10

If that works, you can start adding the other processing settings (e.g. tile sizes) one by one and see if we can narrow this down.

estebanaguilera · January 12, 2021, 6:45pm

Also, I wonder what tile sizes have been set up in the SNAP UI. In order to get this, go to Tools -> Options -> Performance and check Tile size (px).

AriJeannin · January 13, 2021, 9:42am

SNAP UI got 512px Tile size. I changed it for 4096px and:
!

2nd point:
$ gpt DinSAR_chainv0.xml -c 16G -q 10 runned well!

I tried $ gpt DinSAR_chainv0.xml -c 47G -q 14: still good!

512px ok
1024px ok
2048px ok
4096px not ok

What should I understand with this Tile size (px)?

Thanks a lot

estebanaguilera · January 13, 2021, 10:12am

Thanks for testing and great to hear that it works. I’d have some more questions:

When do you get the java.lang.NullPointerException? After updating the tile size setting in the SNAP UI? or after running the graph (from the UI) with the new settings?
If you write to disk after applying the TOPSAR-Split operator (per input product). What are the resulting image dimensions of the master and slave? You can check this in the *.data/*.hdr files relative to the output product directory. See samples and lines. I wonder if there’s a mismatch between master and slave dimensions and whether the images are smaller than the tile size.

AriJeannin · January 13, 2021, 10:52am

I get it during the graph process. Seems that whatever the number I update, there is no messages after changing settings.
Only 2 bursts:

Slave:             | Master:
samples = 22849    | samples = 22849
lines = 3016       | lines = 3016

I tried with 10bursts(lines = 15080) thinking that I can then increase the Tile size to 4096px, but ESD estimates overlaping for each burst:

INFO: org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp: Estimating azimuth offset for blocks in overlap: 1/9
Arrays must have the same length.
Arrays must have the same length.
...

So it seems that it’s not possible to go beyond 3016px Tile size for applying ESD to a S1product?!
Thanks a lot!

estebanaguilera · January 13, 2021, 12:40pm

Thanks for the additional effort.

I cannot put my finger on it yet. It could also be related to the JAI framework.

When the ESD operator runs, the method computeTileStack() is called, which in turn calls:

estimateAzimuthOffset()
performRangeAzimuthShift()

Both methods request tiles:

estimateAzimuthOffset() requests tiles that intersect the overlaps (see here).
performRangeAzimuthShift() requests tiles for all the bursts (see here).

In any case, my suggestion is to use small tiles to minimize unnecessary data loading when overlap data is requested (note that the full tile intersecting the overlap will be loaded, and what falls outside the overlap is discarded).

I also see you set tileWidth, tileHeight (I think you have a typo there), and TileSize when calling GPT. Is there are reason for not just using TileSize? I’m not sure what the difference is.

AriJeannin · January 13, 2021, 2:03pm

Thank you for the accurate documentation!

tileWidth & tileHeight seems indeed to be doublon with TileSize, I just made it that way to be sure.

my calling function is:

    gpt_cli = ['gpt',
               type_utils.to_cmd_string(graph_path),
               '-q', MAX_CORES,  # Maximum parallelism
               '-J-Xms2G -J-Xmx{}'.format(bytes2snap(MAX_MEM)),  # Initially/max allocated memory
               '-J-Dsnap.log.level=WARNING',
               '-J-Dsnap.jai.defaultTileSize={}'.format(TILE_SIZE),  # Tile size, set to 4096
               '-J-Dsnap.dataio.reader.tileWidth={}'.format(TILE_SIZE),
               '-J-Dsnap.dataio.reader.tileHeigh={}'.format(TILE_SIZE),
               '-J-Dsnap.jai.prefetchTiles=true',
               '-c {}'.format(bytes2snap(0.75 * MAX_MEM)),  # Tile cache, up to 75% of max memory
               # '-x',  # Clears the internal tile cache after writing a complete row to the target file
               *other_args]

Very glad to be able to use ESD properly! thanks again!

Martini · January 24, 2021, 2:19pm

sorry during the co-registration process on the linux terminal I had this problem, I am inexperienced on these things, can anyone help me? @AriJeannin @estebanaguilera

AriJeannin · January 25, 2021, 8:55am

ESD operator cannot compute the coreg between pixels that are not similar.
I understood that sometimes it does not work due to low coherence between pairs (forest/bushs/water…).

ps: but the graph should skipp it and continue the workflow

Martini · January 25, 2021, 9:26am

so what do you advise me to do?

mengdahl · January 25, 2021, 9:50am

A 4096 x 4096 tile is 64 times the size of the 512 pix tile - I’m not surprised it generates an error.