ESD issue with GPT

Microwave Toolbox
1

Hi dears,

I wonder if you are able to run the Enhanced Spectral Diversity (ESD) with gpt commands ?

I’m running the following snapshot with Snap interface: no problem! .

image

But when I run the xml (coming from the save of this graph) with gpt command line, It crashes as follow:

Executing processing graph
..13%..27%..40%...54%..67%..81%.-- org.jblas INFO Starting temp DLL cleanup task.
-- org.jblas INFO Deleted 12 unused temp DLL libraries from C:\Users\ajeannin\AppData\Local\Temp
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
Waiting thread received a null tile.
. done.
org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.core.gpf.graph.GraphProcessor$GPFImagingListener.errorOccurred(GraphProcessor.java:379)
        at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(Unknown Source)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at javax.media.jai.PlanarImage.getExtendedData(Unknown Source)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:447)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:479)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computePartialTile(InterferogramOp.java:1365)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computeTileStackForTOPSARProduct(InterferogramOp.java:1290)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computeTileStack(InterferogramOp.java:863)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.cobbleFloat(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
        at org.esa.s1tbx.sentinel1.gpf.TOPSARDeburstOp.computeTileInOneSwathFloat(TOPSARDeburstOp.java:904)
        at org.esa.s1tbx.sentinel1.gpf.TOPSARDeburstOp.computeTileStack(TOPSARDeburstOp.java:811)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.cobbleFloat(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.datamodel.Band.readRasterData(Band.java:309)
        at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataRegion(ProductSubsetBuilder.java:393)
        at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataImpl(ProductSubsetBuilder.java:330)
        at org.esa.snap.core.dataio.AbstractProductReader.readBandRasterData(AbstractProductReader.java:277)
        at org.esa.snap.core.gpf.common.SubsetOp.computeTile(SubsetOp.java:297)
        at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
        at javax.media.jai.SourcelessOpImage.computeTile(Unknown Source)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:116)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at com.sun.media.jai.util.RequestJob.compute(Unknown Source)
        at com.sun.media.jai.util.WorkerThread.run(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.computeTileStack(SpectralDiversityOp.java:827)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        ... 48 more
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzimuthOffset(SpectralDiversityOp.java:1243)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.computeTileStack(SpectralDiversityOp.java:809)
        ... 50 more
Caused by: java.lang.NullPointerException
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzimuthOffset(SpectralDiversityOp.java:1117)
        ... 51 more

Error: java.lang.NullPointerException

(same inputs/subswaths/bursts/WKT/sameeverything)
my spec:

-q 14 -J-Xms2G -J-Xmx63G -J-Dsnap.log.level=WARNING -J-Dsnap.jai.defaultTileSize=4096 -J-Dsnap.dataio.reader.tileWidth=4096 -J-Dsnap.dataio.reader.tileHeigh=4096 -J-Dsnap.jai.prefetchTiles=true -c 47G

For info: when I run without ESD it’s working.

I saw that:

But my graphic interface is on windows, my command line is calling windows gpt with anaconda prompt(linux). So I guess updating snap soft is also updating gpt command ?

If you have any idea, could be great :slight_smile:
Thank you!
Ari

2

how many bursts did you select in TOPS Split? ESD is only required when you use more than one burst.

3

I’m using 3 bursts.

4

probably also related to the SRTM error we experience at the moment. SRTM ZIP-files are corrupted or not found

Can you try to manually download a DEM and use it as input?

5

Ok, I did with eu-dem as my AOI is in europe.

the error is still the same…

6

just to go sure: Did you store it as GeoTiff projected in WGS84?

7

It was EPSG: 4258, I changed it for: 4326 (WGS84) and still the same error.

When I run without ESD it’s working, the DEM is used by “Back-geocoding”, so I don’t really get how the DEM could be the problem?

8

you’re right, the DEM should not be the problem then…
I know that since SNAP 8 some parameters were changed in the ESD operator. Could it be that the graph was created in an earlier version and you are now using SNAP 8?

9

I’m using Snap8, and the graph is generated through it.
With GPT I just launch the xml generated by saving the previous graph, so it should be the same.

Is that possible that GPT of snap and Snap8 interface are not using the same version ?

10

Only if you had previous versions of SNAP installed and the old one is still in your system (e.g. you have gpt.exe twice)

Are both products covering the same area? Sometimes bursts are shifted along track, especially when you combine Sentinel-1A and 1B

11

Ok I got it once, good.

Yes, they are two S1A, the process works with the interface without any warning.

12

in your previous topic the error message was also displayed in the GUI: ESD issue: "Arrays must have the same length."
What has changed since then?

13

Actually not… I just managed to use subset, and it was working with interface only.
(Here I’am with other datas)

14

can’t think of a reason why this happened.

In case you inserted the subset operator into the graph - it could be worth a try to re-create the graph from scratch and only connect to the write operator once all steps are included. Sometimes editing existing graphs caused problems in the past.

15

I did, took all bursts and it took 680 minutes (vs 5min with 3bursts). Works without any problem with interface.
Then used the same graph launched with GPT, same error as previous, but much more “Arrays must have the same length.” messages (see below)

I just guess that there is something different between interface and gpt, can’t think what neither, maybe new version issue… could be good to report to snap’s devs?

Anyway, thanks for your time!

Executing processing graph
...10%...20%...31%..-- org.jblas INFO Starting temp DLL cleanup task.
-- org.jblas INFO Deleted 4 unused temp DLL libraries from C:\Users\ajeannin\AppData\Local\Temp
SEVERE: com.sun.xml.bind.v2.util.XmlFactory: null
org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
        at org.apache.xerces.parsers.AbstractSAXParser.setFeature(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl.setFeatures(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParserImpl(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserFactoryImpl.setFeature(Unknown Source)
        at com.sun.xml.bind.v2.util.XmlFactory.createParserFactory(XmlFactory.java:106)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.getXMLReader(UnmarshallerImpl.java:124)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:214)
        at it.geosolutions.imageioimpl.plugins.tiff.gdal.GDALMetadataParser.parse(GDALMetadataParser.java:60)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.initializeFromMetadata(TIFFImageReader.java:1337)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.seekToImage(TIFFImageReader.java:865)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.seekToImage(TIFFImageReader.java:822)
        at it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReader.getWidth(TIFFImageReader.java:1080)
        at org.esa.snap.dataio.geotiff.GeoTiffImageReader.getImageWidth(GeoTiffImageReader.java:160)
        at org.esa.snap.dataio.geotiff.GeoTiffProductReader.readProduct(GeoTiffProductReader.java:230)
        at org.esa.snap.dataio.geotiff.GeoTiffProductReader.readProductNodesImpl(GeoTiffProductReader.java:136)
        at org.esa.snap.core.dataio.AbstractProductReader.readProductNodes(AbstractProductReader.java:178)
        at org.esa.snap.engine_utilities.gpf.CommonReaders.read(CommonReaders.java:80)
        at org.esa.snap.engine_utilities.gpf.CommonReaders.readCommonProductReader(CommonReaders.java:59)
        at org.esa.snap.engine_utilities.gpf.CommonReaders.readProduct(CommonReaders.java:33)
        at org.esa.snap.dem.dataio.FileElevationModel.init(FileElevationModel.java:53)
        at org.esa.snap.dem.dataio.FileElevationModel.<init>(FileElevationModel.java:49)
        at org.esa.s1tbx.sentinel1.gpf.BackGeocodingOp.getElevationModel(BackGeocodingOp.java:536)
        at org.esa.s1tbx.sentinel1.gpf.BackGeocodingOp.computeTileStack(BackGeocodingOp.java:486)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.getComplexDoubleMatrix(SpectralDiversityOp.java:992)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.getFineOffsets(SpectralDiversityOp.java:968)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzRgOffsets(SpectralDiversityOp.java:958)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.access$200(SpectralDiversityOp.java:110)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp$2.process(SpectralDiversityOp.java:1437)
        at org.esa.snap.core.util.ThreadRunnable.run(ThreadRunnable.java:23)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays must have the same length.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
Arrays of the same length are expected.
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
Waiting thread received a null tile.
90% done.
org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.core.gpf.graph.GraphProcessor$GPFImagingListener.errorOccurred(GraphProcessor.java:379)
        at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(Unknown Source)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at javax.media.jai.PlanarImage.getExtendedData(Unknown Source)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:447)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:479)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computePartialTile(InterferogramOp.java:1365)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computeTileStackForTOPSARProduct(InterferogramOp.java:1290)
        at org.esa.s1tbx.insar.gpf.InterferogramOp.computeTileStack(InterferogramOp.java:863)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.cobbleFloat(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.Operator.getSourceTile(Operator.java:459)
        at org.esa.s1tbx.sentinel1.gpf.TOPSARDeburstOp.computeTileInOneSwathFloat(TOPSARDeburstOp.java:904)
        at org.esa.s1tbx.sentinel1.gpf.TOPSARDeburstOp.computeTileStack(TOPSARDeburstOp.java:811)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.cobbleFloat(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.datamodel.Band.readRasterData(Band.java:309)
        at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataRegion(ProductSubsetBuilder.java:393)
        at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataImpl(ProductSubsetBuilder.java:330)
        at org.esa.snap.core.dataio.AbstractProductReader.readBandRasterData(AbstractProductReader.java:277)
        at org.esa.snap.core.gpf.common.SubsetOp.computeTile(SubsetOp.java:297)
        at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:82)
        at javax.media.jai.SourcelessOpImage.computeTile(Unknown Source)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at javax.media.jai.PlanarImage.getData(Unknown Source)
        at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:449)
        at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java:435)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:116)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
        at javax.media.jai.OpImage.getTile(Unknown Source)
        at com.sun.media.jai.util.RequestJob.compute(Unknown Source)
        at com.sun.media.jai.util.WorkerThread.run(Unknown Source)
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.computeTileStack(SpectralDiversityOp.java:827)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTileStack.java:122)
        at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTileStack.java:86)
        ... 48 more
Caused by: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException
        at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:440)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzimuthOffset(SpectralDiversityOp.java:1243)
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.computeTileStack(SpectralDiversityOp.java:809)
        ... 50 more
Caused by: java.lang.NullPointerException
        at org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp.estimateAzimuthOffset(SpectralDiversityOp.java:1117)
        ... 51 more

Error: java.lang.NullPointerException
16

@estebanaguilera can you please check if you find a bug here?

2 Likes
17

Any update on this? I get the same error message even with a basic graph.

1 Like
18

Would you mind sharing the graph file (xml) and the name of the input products? Feel free to anonymize the paths. I can then try to reproduce/troubleshoot.

2 Likes
19

Hi, of course, please find here an example that is working on SNAP interface but not through GPT.

inSAR_chainv0.xml (6.7 KB)

Thanks for your time!

20

Thanks for the xml file. I ran it from both the SNAP UI and the command line and both finished successfully. I executed:

$ gpt DinSAR_chainv0.xml -c 16G -q 10

Could you maybe share the contents of your gpt.vmoptions file? and the exact command you use for running GPT?