Stacks of interferograms with or without same reference?

I have 35 EnviSAT images and I would like to create some interferograms to see seasonal patterns. I don’t need each combination and it would be computationally very expensive, then I am doing stacks by year, tho I want to merge these stacks later to perform time series. I am using the same DEM for all the stacks.

I was using a reference image for each year, (i.e., an image from Jan/2004 for the stack of 2004, an image from Jan/2005 for the stack of 2005…). I was wondering if I want to merge all my interferograms in a unique stack, should all be referenced to the same image? If that is the case, if I use the Jan/2004 image as the reference for the stacks of 2005, 2006…2010 are there going to be problems related to temporal correlation?