You are right - the polarization does matter in case of backscatter intensity, because cross polarization (VH or HV) is often higher in areas with voluminous objects (branches, canopies, ect.). But the interferogram is based on the phase difference between two images. The phase is the state of the wave when hitting a target. So both VV and VH will hit the target (probably slightly different as you said), but the state of the wave is almost the same.
But I am always open to references which test this difference.
As you see in your interferogram, the only usable areas are along the coast where vegetation cover is low. The areas where you see these fringes (color patterns) are usable for interferometry, but unfortunately, most of your study area lacks of signal correlation. Consequently, there is nothing to unwrap and height information will not be precise.