Geoid to ellipsoid correction in SRTM for terrain correction


I am using ALOS2, RS2 and TDX data and have noticed differences in terrain correction using the downloaded SRTM 1 second, SRTM 3 second DEMs and an external TanDEM-X DEM (higher resolution and better quality).

I previously thought that the TDX DEM would produce the best results, but the SRTM 3 second is actually the best (features like roads that are straight in real life are mostly straight in the image etc., although there is some warping). SRTM 1 second is not as good. Additionally, neither image is in exactly the right place (10-25m off).
The TDX DEM images actually provides the worst result as the geometric correction is further off (up to 35m in some areas) and some of the features are quite warped.

I understand that in the terrain correction process, the SRTM DEMs have their height corrected (from geoid EGM96 to WGS ellipsoid). The TDX DEM height system is already WGS84 ellipsoid so I am wondering if this correction is applied to external DEMs as well (in this case unnecessarily and creating extra error)? If so, can I turn this correction off?

I realise there may be other factors affecting my results. Any help with this would be appreciated as I need to get my images very precisely geolocated for my research.

Thank you,