Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective