Generic Representation Learning

The 2-dimensional embeddings (tSNE) of our representation for MIT places dataset ('library' category) and an unseen subset of our dataset are provided below. The representation organizes the images based on their 3D content (scene layout, relative camera pose to the scene, etc) and independent of their semantics (visible objects, architectural styles) or low-level properties (color, texture, etc). This suggests that the representation must have a notion of certain basic 3D concepts, though it was never provided with an explicit supervision for such tasks (especially for non-matching images, while all tSNE images are non-matching). The tSNE of our dataset also suggests the patches are organized based on their coarse surface normals (again, a task that the representation didn't receive a supervision for). See the section below for quantitative evaluation of our representation for surface normal estimation on NYUv2 dataset.