Artificial Intelligence (AI) is at a crucial point in its development: stable enough to be used in production systems, and increasingly pervasive in our lives. What does that mean for its safety? In his book Normal Accidents, the sociologist Charles Perrow proposed a framework to analyze new technologies and the risks they entail. He showed that major accidents are nearly unavoidable in complex systems with tightly coupled components if they are run long enough. In this essay, we apply and extend Perrow’s framework to AI to assess its potential risks. Today’s AI systems are already highly complex, and their complexity is steadily increasing. As they become more ubiquitous, different algorithms will interact directly, leading to tightly coupled systems whose capacity to cause harm we will be unable to predict. We argue that under the current paradigm, Perrow’s normal accidents apply to AI systems and it is only a matter of time before one occurs. This article appears in the AI & Society track.

The Tesla Model S is an extraordinary machine. As part of my research into the regulation of self-driving cars, I've had the privilege of driving one. Or more accurately, I've had the privilege of being driven by one. On a Colorado highway in July, with some trepidation, I flicked the lever to engage Autopilot mode. I told the representative from Tesla that I was worried about handing over control, taking my feet off the pedals and my hands off the wheel.

When I read the headlines Friday about the fatal crash of a Tesla vehicle while in self-driving mode, I immediately thought about Three Mile Island. It's not that Tesla's autopilot mode is the vehicular equivalent of a nuclear meltdown. As the company would very much like you to note, self-driving cars are doing better, statistically speaking, than human drivers. Tesla says autopilot was used for 130 million miles worth of driving before this fatal crash. Human-driven cars in the U.S. have 1.08 fatal crashes for every 100 million miles.