Exploring 3D-aware Latent Spaces for Efficiently Learning Numerous Scenes