Reducing the Scope of Language Models with Circuit Breakers

Language models are now deployed in a wide variety of user-facing applications, often for specific purposes like answering questions about documentation or acting as coding assistants. As these models are intended for particular purposes, they should not be able to answer irrelevant queries like requests for poetry or questions about physics, or even worse, queries that can only be answered by humans like sensitive company policies. Instead we would like them to only answer queries corresponding to desired behavior and refuse all other requests, which we refer to as scoping. We find that, despite the use of system prompts, two representative language models can be poorly scoped and respond to queries they should not be addressing. We then conduct a comprehensive empirical evaluation of methods which could be used for scoping the behavior of language models. Among many other results, we show that a recently-proposed method for general alignment, Circuit Breakers (CB), can be adapted to scope language models to very specific tasks like sentiment analysis or summarization or even tasks with finer-grained scoping (e.g. When compared to standard methods like fine-tuning or preference learning, CB is more robust both for out of distribution tasks, and to adversarial prompting techniques. We also show that layering SFT and CB together often results in the best of both worlds: improved performance only on relevant queries, while rejecting irrelevant ones. In the past few years Large Language Models have exploded into the popular conscience. One major recent addition is the "alignment" process through Reinforcement Learning with Human Feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022), which has made the current generation of language models much less likely to emit toxic content than previous generations (Wolf et al., 2017), and thus much more acceptable for general use. As a result, many businesses and individuals feel more comfortable using these technologies than they would be in the past. As a result, we have generally capable language models which refuse to answer toxic or dangerous queries, but it is still difficult to deploy these language models. Even though they may not emit toxic content as often, they still will happily answer any question, irrelevant or not. This becomes a problem when we wish to deploy language models as products in specific contexts: e.g. While language models have general language capability, there is still a need to scope them for specific uses. David Yunis is a PhD student at the Toyota Technological Institute at Chicago. Work was performed during an internship at IBM. Arrows indicate the direction of best performance.