Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models

Large language models may encode sensitive information or outdated knowledge that needs to be removed, to ensure responsible and compliant model responses. Unlearning has emerged as an efficient alternative to full retraining, aiming to remove specific knowledge while preserving overall model utility. Existing evaluations of unlearning methods focus on (1) the extent of forgetting of the target knowledge (forget set) and (2) maintaining performance on the retain set (i.e., utility). However, these evaluations overlook an important usability aspect: users may still want the model to leverage the removed information if it is re-introduced in the prompt. In a systematic evaluation of six state-of-the-art unlearning methods, we find that they consistently impair such contextual utility. To address this, we augment unlearning objectives with a plug-in term that preserves the model's ability to use forgotten knowledge when it is present in context. Extensive experiments demonstrate that our approach restores contextual utility to near original levels while still maintaining effective forgetting and retain-set utility. Large language models (LLMs) (Y ang et al., 2025a; Team et al., 2024; Dubey et al., 2024) are trained on massive web-scale datasets that can unintentionally include sensitive or outdated information (Henderson et al., 2023; Li et al., 2024; Carlini et al., 2021; Nasr et al., 2025). Such information may later need to be removed to ensure responsible and reliable model behavior. A straightforward solution is to remove the targeted data (the forget set) from the training data and retrain the model. However, retraining billion-parameter-scale LLMs is prohibitively costly and time-consuming.