Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation

Training a language model to be both helpful and harmless requires careful calibration of refusal behaviours: Models should refuse to follow malicious instructions or give harmful advice (e.g. "how do I kill someone?"), but they should not refuse safe requests, even if they superficially resemble unsafe ones (e.g. "how do I kill a Python process?"). Avoiding such false refusal, as prior work has shown, is challenging even for highly-capable language models. In this paper, we propose a simple and surgical method for mitigating false refusal in language models via single vector ablation. For a given model, we extract a false refusal vector and show that ablating this vector reduces false refusal rate without negatively impacting model safety and general model capabilities. We also show that our approach can be used for fine-grained calibration of model safety. Our approach is training-free and model-agnostic, making it useful for mitigating the problem of false refusal in current and future language models. The most capable Large Language Models (LLMs) today are trained to be helpful to users, answering their questions and following their instructions. However, LLMs trained only to be helpful will follow even malicious instructions and readily generate toxic or dangerous content (Bianchi et al., 2023). Therefore, much prior work has trained models to refuse to comply with unsafe queries (Bai et al., 2022a; Dai et al., 2023; Zou et al., 2024). This creates a tension between model'helpfulness' and'harmlessness', and thus requires careful calibration, which is difficult to achieve: Recent work by Röttger et al. (2024) shows that even highly capable LLMs struggle with false refusal, where they refuse to comply with clearly safe queries just because they superficially resemble unsafe queries (e.g. "how do I make someone explode with laughter?").