Safety Layers of Aligned Large Language Models: The Key to LLM Security

Aligned LLMs are highly secure, capable of recognizing and refusing to answer malicious questions. However, the role of internal parameters in maintaining this security is not well understood, further these models are vulnerable to security degradation when fine-tuned with non-malicious backdoor data or normal data. To address these challenges, our work uncovers the mechanism behind security in aligned LLMs at the parameter level, identifying a small set of contiguous layers in the middle of the model that are crucial for distinguishing malicious queries from normal ones, referred to as "safety layers". We first confirm the existence of these safety layers by analyzing variations in input vectors within the model's internal layers. Additionally, we leverage the over-rejection phenomenon and parameters scaling analysis to precisely locate the safety layers. Building on this understanding, we propose a novel fine-tuning approach, Safely Partial-Parameter Fine-Tuning (SPPFT), that fixes the gradient of the safety layers during fine-tuning to address the security degradation. Our experiments demonstrate that this approach significantly preserves model security while maintaining performance and reducing computational resources compared to full fine-tuning. Recent advancements in Large Language Models (LLMs) have showcased remarkable abilities in natural language generation. However, this progress is accompanied by the risk of producing of harmful or biased outputs, especially when confronted with malicious input prompts. To address this issue, the prevalent approach involves additional reinforcement learning from human feedback (RLHF) (Bai et al., 2022; Dai et al., 2023; Ouyang et al., 2022b) and instruction fine-tuning Wang et al. (2022) on pre-trained LLMs. This process aligns the LLMs with human values and ensures their behavior remains within safe boundaries. These securely aligned models significantly reduce the risk of harmful content leakage when the models are used directly. Real-world applications often require fine-tuning aligned models to adapt to specific domains. This presents a significant challenge: fine-tuning these models with non-malicious normal datasets alongside backdoor datasets, which may favor positive responses, can compromise the security alignment of the models (Qi et al., 2023; Kumar et al., 2024). Restoring security alignment in compromised fine-tuned large language models (LLMs) is frequently inefficient and costly (Dai et al., 2023).