An Essay on Optimization Mystery of Deep Learning

Despite its huge empirical success, deep learning still preserves many features of alchemy [Rahimi, 2017]: progress in this field is obtained mainly by trial and error, and our intuition about how do neural networks actually work often misleads us. Alchemy, in order to become usual chemistry, needs a theoretical ground. For now, a solid theoretical ground for deep learning is lacking, however, fortunately, many pieces of theory appeared from different directions during several past years. The purpose of this essay is not to provide a comprehensive review, but to draw connections between some works on this topic. The list of works mentioned here is by no means representative, or, all the more so, complete. Since the theory of deep learning is lacking, some features of neural networks learning seem "mysterious". We emphasize two mysteries of deep learning: 1. Generalization mystery. It is very common for contemporary neural networks to have many more parameters than the number of training examples at hand.