ESD issue with GPT

Oh okay good that it works! Brings me some hope.

I don’t use the gpt.vmoptions file as I’m using a custom command with a fonction reading number of cores and max memory of the computer that is launching the script.

here with a 16cores / 64gb ram:

gpt DinSAR_chainv0.xml -q 14 -J-Xms2G -J-Xmx63G -J-Dsnap.log.level=WARNING -J-Dsnap.jai.defaultTileSize=4096 -J-Dsnap.dataio.reader.tileWidth=4096 -J-Dsnap.dataio.reader.tileHeigh=4096 -J-Dsnap.jai.prefetchTiles=true -c 47G

ps: still my gpt.vmoptions file

####### Enter one VM parameter per line
####### For example, to adjust the maximum memory usage to 512 MB, uncomment the following line:
####### -Xmx512m
####### To include another file, uncomment the following line:
####### -include-options [path to other .vmoption file]

Could you try running this?

$ gpt DinSAR_chainv0.xml -c 16G -q 10

If that works, you can start adding the other processing settings (e.g. tile sizes) one by one and see if we can narrow this down.

Also, I wonder what tile sizes have been set up in the SNAP UI. In order to get this, go to Tools -> Options -> Performance and check Tile size (px).

SNAP UI got 512px Tile size. I changed it for 4096px and:
image !

2nd point:
$ gpt DinSAR_chainv0.xml -c 16G -q 10 runned well!

I tried $ gpt DinSAR_chainv0.xml -c 47G -q 14: still good!

512px ok
1024px ok
2048px ok
4096px not ok

What should I understand with this Tile size (px)?

Thanks a lot

Thanks for testing and great to hear that it works. I’d have some more questions:

  1. When do you get the java.lang.NullPointerException? After updating the tile size setting in the SNAP UI? or after running the graph (from the UI) with the new settings?
  2. If you write to disk after applying the TOPSAR-Split operator (per input product). What are the resulting image dimensions of the master and slave? You can check this in the *.data/*.hdr files relative to the output product directory. See samples and lines. I wonder if there’s a mismatch between master and slave dimensions and whether the images are smaller than the tile size.
  1. I get it during the graph process. Seems that whatever the number I update, there is no messages after changing settings.

  2. Only 2 bursts:

Slave:             | Master:
samples = 22849    | samples = 22849
lines = 3016       | lines = 3016

I tried with 10bursts(lines = 15080) thinking that I can then increase the Tile size to 4096px, but ESD estimates overlaping for each burst:

INFO: org.esa.s1tbx.sentinel1.gpf.SpectralDiversityOp: Estimating azimuth offset for blocks in overlap: 1/9
Arrays must have the same length.
Arrays must have the same length.

So it seems that it’s not possible to go beyond 3016px Tile size for applying ESD to a S1product?!
Thanks a lot!

Thanks for the additional effort.

I cannot put my finger on it yet. It could also be related to the JAI framework.

When the ESD operator runs, the method computeTileStack() is called, which in turn calls:

  • estimateAzimuthOffset()
  • performRangeAzimuthShift()

Both methods request tiles:

  1. estimateAzimuthOffset() requests tiles that intersect the overlaps (see here).
  2. performRangeAzimuthShift() requests tiles for all the bursts (see here).

In any case, my suggestion is to use small tiles to minimize unnecessary data loading when overlap data is requested (note that the full tile intersecting the overlap will be loaded, and what falls outside the overlap is discarded).

I also see you set tileWidth, tileHeight (I think you have a typo there), and TileSize when calling GPT. Is there are reason for not just using TileSize? I’m not sure what the difference is.

Thank you for the accurate documentation!

tileWidth & tileHeight seems indeed to be doublon with TileSize, I just made it that way to be sure.

my calling function is:

    gpt_cli = ['gpt',
               '-q', MAX_CORES,  # Maximum parallelism
               '-J-Xms2G -J-Xmx{}'.format(bytes2snap(MAX_MEM)),  # Initially/max allocated memory
               '-J-Dsnap.jai.defaultTileSize={}'.format(TILE_SIZE),  # Tile size, set to 4096
               '-c {}'.format(bytes2snap(0.75 * MAX_MEM)),  # Tile cache, up to 75% of max memory
               # '-x',  # Clears the internal tile cache after writing a complete row to the target file

Very glad to be able to use ESD properly! thanks again!

1 Like

sorry during the co-registration process on the linux terminal I had this problem, I am inexperienced on these things, can anyone help me? @AriJeannin @estebanaguilera

ESD operator cannot compute the coreg between pixels that are not similar.
I understood that sometimes it does not work due to low coherence between pairs (forest/bushs/water…).

ps: but the graph should skipp it and continue the workflow

so what do you advise me to do?

A 4096 x 4096 tile is 64 times the size of the 512 pix tile - I’m not surprised it generates an error.