LoCo: Learning 3D Location-Consistent Image Features with a Memory-Efficient Ranking Loss

Image feature extractors are rendered substantially more useful if different views of the same 3D location yield similar features while still being distinct from other locations. A feature extractor that achieves this goal even under significant viewpoint changes must recognise not just semantic categories in a scene, but also understand how different objects relate to each other in three dimensions. Existing work addresses this task by posing it as a patch retrieval problem, training the extracted features to facilitate retrieval of all image patches that project from the same 3D location. However, this approach uses a loss formulation that requires substantial memory and computation resources, limiting its applicability for largescale training. We present a method for memory-efficient learning of locationconsistent features that reformulates and approximates the smooth average precision objective.