Visual Entity Linking: A Preliminary Study

In this paper, we describe a system that jointly extracts entities appearing in images and mentioned in their accompanying captions. As input, the entity linking program takes a segmented image together with its caption. It consists of a sequence of processing steps: part-of-speech tagging, dependency parsing, and coreference resolution that enables us to identify the entities as well as possible textual relations from the captions. The program uses the image regions labelled with a set of predefined categories and computes WordNet similarities between these labels and the entity names. Finally, the program links the entities it detected across the text and the images. We applied our system on the Segmented and Annotated IAPR TC-12 dataset that we enriched with entity annotations and we obtained a correct assignment rate of 55.48%