When AI Systems Fail: Introducing the AI Incident Database - The Partnership on AI

Governments, corporations, and individuals are increasingly deploying intelligent systems to safety-critical problem areas, such as transportation, energy, health care, and law enforcement, as well as challenging social system domains such as recruiting. Failures of these systems pose serious risks to life and wellbeing, but even well-intentioned intelligent system developers fail to imagine what can go wrong when their systems are deployed in the real world. These failures can lead to dire consequences, some of which we've already witnessed, from a trading algorithm causing a market "flash crash" in 2010 to an autonomous car killing a pedestrian in 2018 and a facial recognition system causing the wrongful arrest of an innocent person in 2019. Worse, the artificial intelligence community has no formal systems or processes whereby practitioners can discover and learn from the mistakes of the past, especially since there is not a widely used centralized place to collect information about what has gone wrong previously. Avoiding repeated AI failures requires making past failures known.