Implicit Bias of Gradient Descent on Linear Convolutional Networks

Implicit biases introduced by optimization algorithms play an crucial role in learning deep neural networks [Neyshabur et al., 2015b,a, Hochreiter and Schmidhuber, 1997, Keskar et al., 2016, Chaudhari Consequently, optimization objectives for learning such high capacity models have many global minima that fit training data perfectly. Minimizing the training loss on these models is therefore entirely equivalent to minimizing the training loss for linear classification.