How Do We Know What Superintelligent AI Will Do?

Unpredictability is an intuitively familiar concept. We can usually predict the outcome of common physical processes without knowing specific behavior of particular atoms, just as we can typically predict the overall behavior of the intelligent system without knowing specific intermediate steps. Rahwan and Cebrian observe that "… complex AI agents often exhibit inherent unpredictability: they demonstrate emergent behaviors that are impossible to predict with precision--even by their own programmers. These behaviors manifest themselves only through interaction with the world and with other agents in the environment… In fact, Alan Turing and Alonzo Church showed the fundamental impossibility of ensuring an algorithm fulfills certain properties without actually running said algorithm. There are fundamental theoretical limits to our ability to verify that a particular piece of code will always satisfy desirable properties unless we execute the code, and observe its behavior."