Towards a Theoretical Understanding of Batch Normalization

One of the most important recent innovations for optimizing deep neural networks is Batch Normalization (Bn) [14]. This technique has proved to successfully stabilize and accelerate training of deep neural network and is thus by now standard in many state-of-the art architectures such as ResNets [13] and the latest Inception Nets [29]. The problem addressed by Batch Normalization is the well-known phenomenon of vanishing or exploding gradients that can make training unstable and cause divergence. The root of this problem lies in the use of deep models that involve the composition of nested functions which allows for rich modelling capacity but also creates complex dependencies between the individual parameters. Training such models involves computing gradients as a product of multiple Jacobian matrices which can quickly become unstable if the spectrum of the individual Jacobians is not within an appropriate range. It therefore seems natural that normalizing the inner layers of a neural network might stabilize training which recently led to the development of many such normalization methods such as [2, 3, 15, 27] to name just a few. Despite the key role of Batch Normalization for training deep networks, the community is mostly relying on empirical evidence, lacking a thorough theoretical understanding explaining such success.