Integrating Knowledge and Reasoning in Image Understanding

Deep learning based data-driven approaches have been successfully applied in various image understanding applications ranging from object recognition, semantic segmentation to visual question answering. However, the lack of knowledge integration as well as higher-level reasoning capabilities with the methods still pose a hindrance. In this work, we present a brief survey of a few representative reasoning mechanisms, knowledge integration methods and their corresponding image understanding Figure 1: The diagram shows the information hierarchy for applications developed by various groups images and the knowledge associated with each level of information. of researchers, approaching the problem from a variety of angles. Furthermore, we discuss upon key efforts on integrating external knowledge with neural paper is to present a survey of recent works (including a few networks. Taking cues from these efforts, we of our works) in image understanding where knowledge and conclude by discussing potential pathways to improve reasoning plays an important role.