LAGUNA: LAnguage Guided UNsupervised Adaptation with structured spaces