I tried to do interferometry for a full swath of a TOPSAR pair. I used the standard graph provided with installation and just added enhanced-spectral-diversity after back-geocoing and before interferogram formation into the graph. The processing is fine, and the result looks good but the processing took a lot of time and was finished after a few days even in our good server. The same pair was processed by Gamma or SARscape in about an hour. I wonder if anybody knows a solution for this problem? Is it a problem of SNAP program for processing full-swath interferogram? Is there any way to make the processing much faster? For a burst processing of a single swath using SNAP I did not have such a problem
If you are working on linux you could as well create a shell script which uses gpt and writes intermediate products (memory is then released), such as here: S1_preprocessing.sh (1.0 KB) This is for the preprocessing of a Sentinel-1 product to terrain-corrected Gamma0, but creating an inteferogram should work as well and you can delete intermediate products within the same graph.
Not very elegant and just a work-around.
Slow performance of “complex” graphs is quite an issue if running on cloud instances. With a 4 vCPU and 16 GB RAM VM, I can’t even get beyond the TOPSAR-Split - Calib - TOPSAR-Deburst - TOPSAR-Merge stage (as part of HAAlpha processing). Typically crashes out of memory after some 20% of processing (10 mins). If you htop the process, it seems that it simply continues stuffing everything in memory. Also, it looks the more steps you include in the graph, the worst parallel-CPU use becomes, with most of the processing done by 1 or 2 processors only. This looks a lot like software design issues, but I could be mistaken
I am switching to step-wise processing with intermediate storage for now. Getting larger VMs would be an option, but prices scale quickly with more exotic configs. Disk space is less costly.
The full process is in TOPSAR_SLC_HAAlpa.xml but that never completes.
I have split into several steps now. The first one (step1.sh) does split-calib-deburst and writes out each subswath. That completes in about 20 mins.
The second one (step2.sh) does TOPSAR-Merge and Multilook (only) and takes > 90 mins, for most of which the CPUs appear to be idle. Same if I only do TOPSAR-Merge (which seems to be the real bottleneck). I would expect TOPSAR-Merge to be some simple read - combine - write operation (maybe with some interpolation).
We’ve looked at your graph and it appears to be unnecessary complex whick kills the performance. Calibration is able to handle bursts and subswaths so it’s not necessary to split, deburst, calibrate and then merge. @lveci has more details.
However, the terrain correction also has a problem causing it to get a lot of source tiles. This has been fixed and should be in an update later this month. Currently this graph is taking around 10min to process.
The next bottleneck and memory consumption to tackle is in the read.
Bigger VM is not an option. There is no logical reason why this graph does not run in 16 GB RAM, other than bad memory management. I’ll stick to the split - merge 3 step approach for now and wait for further updates on memory mgmt.