the vertical displacement usually is just a relative measure. So you should then level it to your ground by identifying a pixel where you expect no change and subtract this value from your image.
You could also consider to mask out areas with low coherence.
Both was described in this topic: