The impossibility of intelligence explosion – François Chollet – Medium

In 1965, I. J. Good described for the first time the notion of "intelligence explosion", as it relates to artificial intelligence (AI): Decades later, the concept of an "intelligence explosion" -- leading to the sudden rise of "superintelligence" and the accidental end of the human race -- has taken hold in the AI community. Famous business leaders are casting it as a major risk, greater than nuclear war or climate change. Average graduate students in machine learning are endorsing it. In a 2015 email survey targeting AI researchers, 29% of respondents answered that intelligence explosion was "likely" or "highly likely". A further 21% considered it a serious possibility. The basic premise is that, in the near future, a first "seed AI" will be created, with general problem-solving abilities slightly surpassing that of humans. This seed AI would start designing better AIs, initiating a recursive self-improvement loop that would immediately leave human intelligence in the dust, overtaking it by orders of magnitude in a short time.