Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion

--With the increasing multimodal knowledge privatization requirements, multimodal knowledge graphs in different institutes are usually decentralized, lacking of effective collaboration system with both stronger reasoning ability and transmission safety guarantees. In this paper, we propose the Federated Multimodal Knowledge Graph Completion (FedMKGC) task, aiming at training over federated MKGs for better predicting the missing links in clients without sharing sensitive knowledge. We propose a framework named MMFeD3-HidE for addressing multimodal uncertain unavailability and multimodal client heterogeneity challenges of FedMKGC. We propose a FedMKGC benchmark for a comprehensive evaluation, consisting of a general FedMKGC backbone named MMFedE, datasets with heterogeneous multimodal information, and three groups of constructed baselines. Experiments conducted on our benchmark validate the effectiveness, semantic consistency, and convergence robustness of MMFeD3-HidE. Multimodal knowledge graphs (MKGs) [1], [2] organize graph structures composed of relational triples (head entity, relation, tail entity) and their visual and textual attributes as Figure 1, which have been widely in multimodal knowledge-intensive tasks [3]-[6]. Due to incomplete construction and new knowledge emergence, Multimodal Knowledge Graph Completion (MKGC) task [7], [8] has been widely-explored [9]-[12] to reason the missing links in MKGs with multimodal information. For example, in Figure 1, given a query (Kobe Bryant, T eam member,?), MKGC aims to predict the tail entity L.A. Lakers with graph structures, entity images, and descriptions. In real world, the MKGs are usually decentralized in different institutes due to commercial interests or data regulations, such as open-sourced MKGs DBpedia [13], Wiki-Ying Zhang, Y u Zhao, Xuhui Sui, Baohang Zhou, Xiangrui Cai, Xiaojie Y uan are with the College of Computer Science, VCIP, TMCC, TBI Center, DISSec, Nankai University, Tianjin, China (e-mail: yingzhang@nankai.edu.cn, Li Shen is with the Shenzhen Campus of Sun Y at-sen University, Shenzhen, China (e-mail: mathshenli@gmail.com).