VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models