Replay to Remember: Retaining Domain Knowledge in Streaming Language Models

Traditional fine-tuning methods, while effective, often require substantial computational resources and large, static datasets, making them impractical for real-time applications. Moreover, these models notoriously suffer from catastrophic forgetting, rapid performance degradation on previously learned tasks when presented with new data (Luo et al., 2023). Recent literature addresses catastrophic forgetting via techniques such as replay buffers, which periodically reintroduce previously learned data, and Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning approach designed to reduce computational overhead (Smith & Jones, 2024; Hu et al., 2021). Although these methods individually show promise, there remains a notable gap in understanding their efficacy and interaction within real-time, streaming learning environments. In this work, we bridge this gap by integrating LoRA with a lightweight replay mechanism under stringent streaming constraints, simulating real-world conditions where models must continually adapt using limited computational resources and data batches. We focus specifically on three distinct domains,medical, genetic, and legal,to evaluate the generalizability and robustness of our approach.