Why does deep learning work so well?

Modern machine learning research has demonstrated remarkable achievements. Today, we can train machines to detect objects in images, extract meaning from text, stop spam emails, drive cars, discover new drug candidates, and beat top players in Chess, Go, and countless other games. A lot of these advancements are powered by deep learning, in particular deep neural networks. Yet, the theory behind deep neural networks remains poorly understood. Sure, we understand the math of what individual neurons are doing, but we're lacking a mathematical theory of the emergent behavior of entire network.