Seeing the Bigger Picture: 3D Latent Mapping for Mobile Manipulation Policy Learning