Two Computer Doomsday Scenarios: How Likely Are They?

In an open-access paper last year at the Journal of Artificial Intelligence Research, a research group concluded that a computer superintelligence, if developed, could not be contained. It would be a HAL 9000 that couldn't just be turned off. The catch is that controlling a super-intelligence far beyond human comprehension would require a simulation of that super-intelligence which we can analyze (and control). But if we're unable to comprehend it, it's impossible to create such a simulation. Rules such as'cause no harm to humans' can't be set if we don't understand the kind of scenarios that an AI is going to come up with, suggest the authors of the new paper.