AI Alarmism: why smart people believe dumb things about our future AI overlords

Maciej Cegłowski (previously) gave this talk, "Superintelligence: The Idea That Eats Smart People," at Web Camp Zagreb last October, spending 45 minutes delving into the origin of the idea that computers are going to become apocalyptic, self-programming, superintelligent basilisks that end all live on Earth (and variations on this theme) and then explaining why this fundamentally evidence-free, fuzzy idea has colonized so many otherwise brilliant people -- including people like Stephen Hawking -- and why it's an irrational and potentially harmful belief system. As a science fiction writer, I've spent a fair bit of time noodling with these ideas in both story form and essays: True Names, the novella I wrote with Ben Rosenbaum about this, was nominated for a Hugo Award; and Charlie Stross and I wrote a novel on the theme, Rapture of the Nerds, and then there's my essay on the Singularity as a spiritual belief system that can pass for a scientific prediction. One thing I'm keenly aware of is that the aesthetic appeal of futuristic Singularity predictions is firmly rooted in the here-and-now: it's nice to think that there is a thing called "progress," and that we're in the midst of it; it's nice to think that when progress outstrips your capacity to make sense of it, it's because it's transcended human comprehension (and not, say, because your time has past and you are becoming irrelevant to a field and discourse you once dominated); it's nice to think that the privilege you enjoy in the midst of great deprivation is in the service to a better future for all humanity, and not a fundamentally unfair situation that you would rise up in fury over if the roles were reversed. Cegłowski's expert puncturing of the arguments for "AI Alarmism" was prompted by philosopher Nick Bostrom's bestselling book Superintelligence: Paths, Dangers, Strategies, which is a fun read but which also palms a lot of cards in the construction of its arguments. I believe the greater social meaning of AI Alarmism is the twin phenomena of the worry of an ever-larger class of have-nots whose lives are upended by uneven economic returns from technological disruption (in other words, the problem isn't that only some of us have to clean toilets, while all of us have to use them; it's that the dividends from self-cleaning toilets never accrue to the toilet-cleaners they displace); and the blithe dismissal of this worry by an ever-smaller, ever-richer 1%, who use the story of AI as a spiritual belief system that declares this division to be natural, inevitable, and, ultimately, beneficial.