A Definition of a batch normalization layer

A small constant is included in the denominator for numerical stability. For distributed training, the batch statistics are usually estimated locally on a subset of the training minibatch ("ghost batch normalization" [32]). In figure 2 of the main text, we studied the variance of hidden activations and the batch statistics of residual blocks at a range of depths in three different architectures; a deep linear fully connected unnormalized residual network, a deep linear fully connected normalized residual network and a deep convolutional normalized residual network with ReLUs. We now define the three models in full. Deep fully connected linear residual network without normalization: The inputs are 100 dimensional vectors composed of independent random samples from the unit normal distribution, and the batch size is 1000.