you should also consider including the coherence. Only areas with coherence > 0.4 can be used. Maybe the blue ones are of low coherence and therefore invalid.
You can define a valid pixel expression as suggested here:
Besides that, there can still be atmospheric disturbances left which could affect your fringes.
If you want to go sure, analyze more image pairs and compare if the patterns of subsidence are always more or less the same. If not, chances are high that some of them are just random noise.