Nobody Knows How to Safety-Test AI

Beth Barnes and three of her colleagues sit cross-legged in a semicircle on a damp lawn on the campus of the University of California, Berkeley. They are describing their attempts to interrogate artificial intelligence chatbots. "They are, in some sense, these vast alien intelligences," says Barnes, 26, who is the founder and CEO of Model Evaluation and Threat Research (METR), an AI-safety nonprofit. "They know so much about whether the next word is going to be'is' versus'was.' We're just playing with a tiny bit on the surface, and there's all this, miles and miles underneath," she says, gesturing at the potentially immense depths of large language models' capabilities. Researchers at METR look a lot like Berkeley students--the four on the lawn are in their twenties and dressed in jeans or sweatpants.