Hi,
I use the latest SNAP 6 on Windows 7, and imported a TanDEM-X product, which I did a subset of, in order to make the processing faster. It is approximately 4000x5000 pixels. Then I created an interferogram (with flat earth and topography flattening, using an external lidar DTM, 2 m pixels), and then I Goldstein filtered this. In order to look at the phase, that it looks reasonable, I had to save it to disk, since “the processing is not performed before it is needed” or something like that. Now this saving to disk has been going on for 14 hours (now at 87%), on an Intel i7, 4.5 GHz, 64 GB Ram, and SSD disks. Something gotta be wrong…
Any suggestions what might cause this extremely slow process?
It may not be writing the disk that is a bottleneck, but rather the “on demand” calculation of values to write.
Is the system sluggish for other tasks? Do you have space on your disk? If there is stuff in the Recycle bin, emptying the Recycle bin might help.
It might be useful to state the value of “Memory” in the “About SNAP” panel.
If you have AV software see if there is a a tool to display status of current scans and check the logs for anything related to your files. I’ve had problems in the past with AV Software failing while scanning large remote sensing data files, giving up, and restarting the scan over and over.
You can poke around in Task Manager to see if anything looks odd: CPU very busy or idle, memory usage, high page faults, etc.
Has the AV software been busy (in Processes tab, select both CPU and CPU Time for viewing)?
Event Viewer may give some hints about the underlying problem.
The system is quick and responsive to everything else. The disk had about 1 Tb free, and the output that now is finished became 339 Mb.
The Memory in About is 39140 Mb.
Unfortunately I cannot turn off the AV, since it is disabled by the administrators (it is a university computer). This could potentially be a reason, although I don’t have any past such experiences on this computer.
CPU is close to 0% before and after the writing, but while it is writing, it was running at ~90% on 1 out of 4 (2 of 8) cores. If I select “save to disk” already in the step of interferogram generation, all cores run at 80%.
The AV has not been busy according to the CPU/CPU Time at least.
The Event Viewer illustrated some error since a couple months with the AV software (which however seems to work), that I will try to solve tomorrow.
Thanks for commenting.
PS. I increased the cache in the SNAP settings from 1024 Mb to 10240 Mb, and the percentages are counting up faster than before (could be a factor 10) - however, the writing (processing on demand) still seems way out of reasonable regarding the time.
So, a restart solved the problem related to the AV software. However, it did not solve the very slow writing to disk by SNAP. This time (after increasing the buffer), the writing to disk took about 18 hours (339 Mb).
I don’t know if this problem is related to this thread, GeoTIFF-BigTIFF writer is very slow when saving metadata
but since it is storing the Meta-data here as well, this might be the problem. However, I don’t if there is a way to disable storing of meta data, when saving in the BEAM-DIMAP format? If i do a new subset of the previous subset, the saving process takes like 10-30 seconds (and the size is about half the previous subset, 18 hours, so the relation is really unreasonable).
So I reply to my own post for documentation.
The slow processing/writing seems to be caused by the resolution of the external DEM. When I go from 10 m resolution to 5 and 2 m for the External DEM, the processing time increases drastically. Even though this should not be such a problem, I guess there is some kind of resampling going on in the background, which causes it.
2 Likes