Analysis on Gradient Propagation in Batch Normalized Residual Networks

We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, inthis work. By analyzing the mean and variance behavior of the input and the gradient in the forward and backward passes through the BN and residual branches, respectively, we show that they work together to confine the gradient variance to a certain range across residual blocks in backpropagation. Asa result, the gradient vanishing/explosion problem is avoided. We also show the relative importance of batch normalization w.r.t. the residual branches in residual networks. Keywords: Batch normalization, Residual network, gradient vanishing/explosion, backpropagation gradient analysis 1. Introduction Convolutional neural networks (CNNs) (LeCun et al., 1989; Bengio et al., 2009; Krizhevsky et al., 2012) aim at learning a feature hierarchy where higher level features are formed by the composition of lower level features. The deep neural networks act as stacked networks with each layer depending on its previous layer's output. The stochastic gradient descent (SGD) method (Simard et al., 1998) has proved to be an effective way in training deep networks. Preprint submitted to Elsevier December 4, 2018 from a given dataset is fed at each training step.