you can mask out low coherence areas in the image using a valid pixel expression as described here: Subsidence map in 3d view) but this unfortunately doesn’t affect the uwnrapping process.
I don’t know how these can be excluded from the unwrapping but the basic algorithms are explained here:
https://ieeexplore.ieee.org/abstract/document/1036000/
Chen and Zebker are the authors of snaphu, unfortunately, the page is currently offline: https://nova.stanford.edusar_group/snaphu/