Why AI Breaks Bad

Once in a while, LLMs turn evil--and no one quite knows why. The AI company Anthropic has made a rigorous effort to build a large language model with positive human values. The $183 billion company's flagship product is Claude, and much of the time, its engineers say, Claude is a model citizen. Its standard persona is warm and earnest. When users tell Claude to "answer like I'm a fourth grader" or "you have a PhD in archeology," it gamely plays along. It makes threats and then carries them out. And the frustrating part--true of all LLMs--is that no one knows exactly why. Consider a recent stress test that Anthropic's safety engineers ran on Claude. In their fictional scenario, the model was to take on the role of Alex, an AI belonging to the Summit Bridge corporation.