Pre-training Transformers for Knowledge Graph Completion

Co-training LMs and KG completion As a fundamental component of human intelligence, models has been shown to be effective in improving relational knowledge plays a crucial role the performance of downstream knowledgeintensive in imitating human cognitive abilities with machine NLP tasks, but not so much for the KG learning (Halford et al., 2010). Knowledge completion task itself (Wang et al., 2021; Yasunaga graphs (KGs) are the most widely used representation et al., 2022). Despite the progress on transferring of relational knowledge, with well-known knowledge between structured KGs and unstructured examples such as Freebase (Bollacker et al., 2008), texts, the generalization from one KG to another YAGO (Suchanek et al., 2007), and Wikidata (Vrandečić is still an open problem that is rarely studied and Krötzsch, 2014). KG is also a key ingredient (Kocijan and Lukasiewicz, 2021).