Maximizing paper clips


In What's the Future, Tim O'Reilly argues that our world is governed by automated systems that are out of our control. Alluding to The Terminator, he says we're already in a "Skynet moment," dominated by artificial intelligence that can no longer be governed by its "former masters." The systems that control our lives optimize for the wrong things: they're carefully tuned to maximize short-term economic gain rather than long-term prosperity. The "flash crash" of 2010 was an economic event created purely by the software that runs our financial systems going awry. However, the real danger of the Skynet moment isn't what happens when the software fails, but when it is working properly: when it's maximizing short-term shareholder value, without considering any other aspects of the world we live in.