Post-Training Language Models for Continual Relation Extraction

Real-world data, such as news articles, social media posts, and chatbot conversations, is inherently dynamic and non-s tationary, presenting significant challenges for constructing real-t ime structured representations through knowledge graphs (KGs). Relation Extraction (RE), a fundamental component of KG creation, often struggl es to adapt to evolving data when traditional models rely on static, out dated datasets. Continual Relation Extraction (CRE) methods tackle this is sue by in-crementally learning new relations while preserving previ ously acquired knowledge. This study investigates the application of pre-trained language models (PLMs), specifically large language models (LL Ms), to CRE, with a focus on leveraging memory replay to address cata strophic forgetting. We evaluate decoder-only models (eg, Mistral-7B and Llama2-7B) and encoder-decoder models (eg, Flan-T5 Base) on the TAC RED and FewRel datasets. Task-incremental fine-tuning of LLMs d emonstrates superior performance over earlier approaches using encode r-only models like BERT on TACRED, excelling in seen-task accuracy and overall performance (measured by whole and average accuracy), part icularly with the Mistral and Flan-T5 models. Results on FewRel are si milarly promising, achieving second place in whole and average accu racy metrics. This work underscores critical factors in knowledge transf er, language model architecture, and KG completeness, advancing CRE wit h LLMs and memory replay for dynamic, real-time relation extracti on.