Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction

Herzig, Roei, Raboh, Moshiko, Chechik, Gal, Berant, Jonathan, Globerson, Amir

Neural Information Processing Systems 

Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple inter-related objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structured prediction, which optimizes over complex labels, while modeling within-label interactions. However, it is unclear what principles should guide the design of a structured prediction model that utilizes the power of deep learning components. Here we propose a design principle for such architectures that follows from a natural requirement of permutation invariance.