PGMEL: Policy Gradient-based Generative Adversarial Network for Multimodal Entity Linking

Abstract--The task of entity linking, which involves associating mentions with their respective entities in a knowledge graph, has received significant attention due to its numerous potential applications. Recently, various multimodal entity linking (MEL) techniques have been proposed, targeted to learn comprehensive embeddings by leveraging both text and vision modalities. The selection of high-quality negative samples can potentially play a crucial role in metric/representation learning. However, to the best of our knowledge, this possibility remains unexplored in existing literature within the framework of MEL. T o fill this gap, we address the multimodal entity linking problem in a generative adversarial setting where the generator is responsible for generating high-quality negative samples, and the discriminator is assigned the responsibility for the metric learning tasks. Since the generator is involved in generating samples, which is a discrete process, we optimize it using policy gradient techniques and propose a policy gradient-based generative adversarial network for multimodal entity linking (PGMEL). Experimental results based on Wiki-MEL, Richpedia-MEL and WikiDiverse datasets demonstrate that PGMEL learns meaningful representation by selecting challenging negative samples and outperforms state-of-the-art methods. The last few decades have seen unprecedented growth in data availability. However, the increasing data availability quickly becomes a liability rather than an asset due to the increased gap between data and information. Thus, information extraction (IE) techniques to retrieve knowledge/information from a large amount of data have received considerable attention recently. A knowledge graph (KG) is a structured information database that allows storing extracted information from a large amount of data for retrieval or reasoning at a later stage. Furthermore, the recent developments in IE techniques allow the automatic creation of large KGs with millions of entries from knowledge bases such as Wikipedia, DBpedia, Freebase, and Y AGO [1]. Automated KG construction is a complex task that involves various intricate subtasks, including (i) named entity recognition to identify and categorize named entities, like a person or geographic locations, etc., in text, (ii) co-reference resolution to group references of the same entity, (iii) relation extraction to establish relationships between the entities, and (iv) entity linking [2], [3]. KM Pooja is with the Department of Information Technology, Indian Institute of Information Technology, Allahabad India 211012. Cheng Long and Aixin Sun are with the School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798.