Unsupervised learning of 3D structure from images

Earlier this week we looked at how deep nets can learn intuitive physics given an input of objects and the relations between them. If only there was some way to look at a 2D scene (e.g., an image from a camera) and build a 3D model of the objects in it and their relationships… Today's paper choice is a big step in that direction, learning the 3D structure of objects from 2D observations. The 2D projection of a scene is a complex function of the attributes and positions of the camera, lights and objects that make up the scene. If endowed with 3D understanding agents can abstract away from this complexity to form stable disentangled representations, e.g., recognizing that a chair is a chair whether seen from above or from the side, under different lighting conditions, or under partial occlusion. Moreover, such representations would allow agents to determine downstream properties of these elements more easily and with less training, e.g., enabling intuitive physical reasoning… The approach described is this paper uses an unsupervised deep learning end-to-end model and "demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner."