A Sober Look at Neural Network Initializations

Improving and understanding the training phase of deep neural networks has attracted a lot of attention in the last couple of years. This training phase mostly consists of minimizing an empirical risk term, and due to the structure of deep neural networks, the corresponding optimization landscape is convoluted and highly non-convex. To avoid getting stuck in local minima several variants of stochastic gradient descent have been proposed and successfully applied. These success stories suggest that the initialization of neural networks, that is, choosing the starting point of the optimization, has become less important. In fact, the two commonly used heuristics proposed in [7, 9] both focus on normalizing the variance of the weights of the neural network to ensure that the gradients of deep networks do not exponentially explode or implode. So far, however, positive or negative side-effects of these initialization strategies have not been investigated in depth. This is the first goal of our paper, and the second goal is to use these insights to develop a new initialization strategy. To be a bit more specific let ·: R [0,) be the ReLU function, that is t: max{0, t}.