Google uses crowdsourced photos to recreate landmarks in 3D for AR/VR

Historically, human artists have been challenged to recreate real-world locations as 3D models, particularly when applications call for photorealistic accuracy. But Google researchers have come up with an alternative that could simultaneously automate the 3D modeling process and improve its results, using a neural network with crowdsourced photos of a location to convincingly replicate landmarks and lighting in 3D. The idea behind neural radiance fields (NeRF) is to extract 3D depth data from 2D images by determining where light rays terminate, a sophisticated technique that alone can create plausible textured 3D models of landmarks. Google's NeRF in the Wild (NeRF-W) system goes further in several ways. First, it uses "in-the-wild photo collections" as inputs, expanding a computer's ability to see landmarks from multiple angles. Next, it evaluates the images to find structures, separating out photographic and environmental variations such as image exposure, scene lighting, post-processing, and weather conditions, as well as shot-to-shot object differences such as people who might be in one image but not another.