simply have a look at the distribution of values in your displacement map. The minimum and maximum values are probably outside the reasonable range.
I just pointed that out because many people are irritated by the range of the resulting displacement values because it is unrealistic (examples here, here and here).
Take this as an example:
It is clear that the actual displacement is not reaching the extremes, although these values were computed by unwrapping. You can use the mask manager to check which areas are affected by setting thresholds along this gradient.
I wouldn’t necessarily remove them but keep in mind that outliers shouldn’t be trusted too much. What matters is if the overall displacement leads to a clear pattern that makes sense.