Scene Graph based Image Retrieval -- A case study on the CLEVR Dataset

The challenges of such a problem involve understanding the nuances of interaction between the multiple modalities, and handling a real-time retrieval from large-scale catalog. While neural models with their rich express-ibility to encode such complex modalities, have revolution-alized research on complex multimodal tasks, the standard practices of end-to-end pure neural style training fails to explicitly model the latent structures present in the different modalities or the different strategies required for the complex task. Without so, the neural model can make blatant mistakes, which its earlier symbolic counterparts would not have made. In this work, we propose a neural symbolic approach for modeling a caption based image retrieval task. The backbone of such a modeling requires a scene-graph representation, [3] of the image catalog and the ongoing di-1 Indian Institute of Technology, Madras 2 IBM Research Labs, Bangalore 3 Indian Institute of Technology, Bombay alog context, and the retrieval task is modeled as a graph subsumption problem.