An Invariant Latent Space Perspective on Language Model Inversion