The Darwinian Argument for Worrying About AI

A broad coalition of AI experts recently released a brief public statement warning of "the risk of extinction from AI." There are many different ways in which AIs might become serious dangers to humanity, and the exact nature of the risks is still debated, but imagine a CEO who acquires an AI assistant. They begin by giving it simple, low-level assignments, like drafting emails and suggesting purchases. As the AI improves over time, it progressively becomes much better at these things than their employees. So the AI gets "promoted."