LLMScan: Causal Scan for LLM Misbehavior Detection

Despite the success of Large Language Models (LLMs) across various fields, their potential to generate untruthful, biased and harmful responses poses significant risks, particularly in critical applications. This highlights the urgent need for systematic methods to detect and prevent such misbehavior. Extensive experiments across various tasks and models reveal clear distinctions in the causal distributions between normal behavior and misbehavior, enabling the development of accurate, lightweight detectors for a variety of misbehavior detection tasks. Large language models (LLMs) demonstrate advanced capabilities in mimicking human language and styles for diverse applications (OpenAI, 2023), from literary creation (Yuan et al., 2022) to code generation (Li et al., 2023; Wang et al., 2023b). At the same time, they have shown the potential to misbehave in various ways, raising serious concerns about their use in critical real-world applications. First, LLMs can inadvertently produce untruthful responses, fabricating information that may be plausible but entirely fictitious, thus misleading users or misrepresenting facts (Rawte et al., 2023). Second, LLMs can be exploited for malicious purposes, such as through jailbreak attacks (Liu et al., 2024; Zou et al., 2023b; Zeng et al., 2024), where the model's safety mechanisms are bypassed to produce harmful outputs. Third, the generation of toxic responses such as insulting or offensive content remains a significant concern (Wang & Chang, 2022). Lastly, biased responses, which can appear as discriminatory or prejudiced remarks, are especially troubling as they have the potential to reinforce stereotypes and undermine societal efforts toward equality and inclusivity (Stanovsky et al., 2019; Zhao et al., 2018). Numerous attempts have been made to detect LLM misbehavior (Pacchiardi et al., 2023; Robey et al., 2024; Sap et al., 2020; Caselli et al., 2021). However, existing approaches often face two significant limitations. First, they tend to focus on a single type of misbehavior, which reduces the overall effectiveness of each method and requires the integration of multiple systems to comprehensively address the diverse forms of misbehavior. Second, many methods rely on analyzing the model's responses, which can be inefficient or even ineffective, particularly for longer outputs. Additionally, they are often vulnerable to adaptive adversarial attacks (Sato et al., 2018; Hartvigsen et al., 2022). As a result, there is an urgent need for more general and robust misbehavior detection methods capable of identifying (and mitigating) the full range of LLM misbehavior. In this work, we introduce LLMScan, an approach designed to address this critical need.