Large Language Model Enhanced Knowledge Representation Learning: A Survey

Large language models (LLMs) (e.g., BERT [18], LLaMA [59]) which represents a direction of ever-increasing model sizes pre-trained on larger corpora, have demonstrated powerful capabilities in solving natural language processing (NLP) tasks, including question answering [99], text generation [100] and document understanding [101]. There are no clear and static thresholds regarding the model sizes. Early LLMs (e.g., BERT, RoBERTa) adopt an encoder architecture and show capabilities in text representation learning and natural language understanding. In recent years, more focus has been given to larger encoder-decoder [102] or decoder-only [103] architectures. As the model size scales up, such LLMs have also shown reasoning ability and even more advanced emergent ability [104], exposing a strong potential for Artificial General Intelligence (AGI). This inflection point, with the arrival of LLMs, marks a paradigm shift from explicit knowledge representation to a renewed focus on the hybrid representation of both explicit knowledge and parametric knowledge. As a popular approach for explicit knowledge representation, KGs are now widely investigated for the combination with Transformer-based LLMs, including pretrained masked language models (PLMs) like BERT and RoBERTa, and more recent generative LLMs like the GPT series and LLaMA. Some works use LLMs to augment knowledge graph representation learning.