Mechanistic Anomaly Detection for "Quirky" Language Models

As LLMs grow in capability, the task of supervising LLMs becomes more challenging. Supervision failures can occur if LLMs are sensitive to factors that supervisors are unaware of. We investigate Mechanistic Anomaly Detection (MAD) as a technique to augment supervision of capable models; we use internal model features to identify anomalous training signals so they can be investigated or discarded. We train detectors to flag points from the test environment that differ substantially from the training environment, and experiment with a large variety of detector features and scoring rules to detect anomalies in a set of "quirky" language models. We find that detectors can achieve high discrimination on some tasks, but no detector is effective across all models and tasks. MAD techniques may be effective in low-stakes applications, but advances in both detection and evaluation are likely needed if they are to be used in high stakes settings. Large language models (LLMs) excel at a wide range of tasks, yet as their capabilities grow, ensuring effective human oversight becomes increasingly challenging. These considerations have motivated Mechanistic Anomaly Detection (MAD): instead of attempting to understand all facets of a model's internal reasoning, MAD focuses on extracting a signal that indicates when an episode is anomalous--flagging behaviors that deviate from the expected patterns without assuming specific trigger mechanisms (Christiano, 2022). This signal could be used to filter out anomalous episodes, or to trigger deeper investigation.