TL;DR: We give a threat model literature review, propose a categorization and describe a consensus threat model from some of DeepMind's AGI safety team. See our post for the detailed literature review. The DeepMind AGI Safety team has been working to understand the space of threat models for existential risk (X-risk) from misaligned AI. Our aim was to clarify the case for X-risk to enable better research project generation and prioritization. First, we conducted a literature review of existing threat models, discussed their strengths/weaknesses and then formed a categorization based on the technical cause of X-risk and the path that leads to X-risk.

Researchers from the University of Oxford and Google's artificial intelligence division DeepMind have claimed that there is a high probability of advanced forms of AI becoming "existentially dangerous to life on Earth". In a recent article in the peer-reviewed journal AI Magazine, the researchers warned that there would be "catastrophic consequences" if the development of certain AI agents continues. Leading philosphers like Oxford University's Nick Bostrom have previously spoken of the threat posed by advanced forms of artificial intelligence, though one of authors of the new paper claimed such warnings did not go far enough.

Disease warfare, leaks of dangerous pathogens from labs and misuse of artificial intelligence could be much more dangerous than the COVID pandemic, a report has warned. The UK government should now be making "serious efforts" to guard against such "extreme risks", say the authors of the Future Proof study. They warn such threats have the potential for huge loss of life globally - and at their worst could even "lead to the premature extinction of humanity". There is a one in six "Russian roulette" chance of an "existential catastrophe over the next one hundred years", according to one of the report's lead authors. Oxford and Cambridge University researchers worked on the study for The Centre for Long-Term Resilience and say it is a "once-in-a-generation opportunity" for the UK to beef up its protections.

Framed in positive terms, this report examines how technical AI research might be steered in a manner that is more attentive to humanity's long-term prospects for survival as a species. In negative terms, we ask what existential risks humanity might face from AI development in the next century, and by what principles contemporary technical research might be directed to address those risks. A key property of hypothetical AI technologies is introduced, called \emph{prepotence}, which is useful for delineating a variety of potential existential risks from artificial intelligence, even as AI paradigms might shift. A set of \auxref{dirtot} contemporary research \directions are then examined for their potential benefit to existential safety. Each research direction is explained with a scenario-driven motivation, and examples of existing work from which to build. The research directions present their own risks and benefits to society that could occur at various scales of impact, and in particular are not guaranteed to benefit existential safety if major developments in them are deployed without adequate forethought and oversight. As such, each direction is accompanied by a consideration of potentially negative side effects.

It has become a cliche to declare that the future is full of both "great promise and great peril." Nonetheless, this aphorism expresses an important fact about the Janus-faced nature of our increasingly powerful technologies. If humanity realizes the best possible future, we could quite possibly usher in an era of unprecedented human flourishing, happiness, and value. But if the great experiment of civilization fails, our species could meet the same fate as the dinosaurs. I find it helpful to think about what a child born today could plausibly expect to witness in her or his lifetime.