The Implicit and Explicit Regularization Effects of Dropout

Dropout is a commonly used regularization technique for neural nets (Hinton et al., 2012; Srivastava et al., 2014). In NLP, dropout is the norm on both small and large models, as it is much more effective than methods such as l 2 regularization (Merity et al., 2017a). In vision, dropout is often used to train extremely large models such as EfficientNet-B7 (Tan & Le, 2019). At training time, dropout sets a random subset of activations to zero, perturbing the network output with a remarkable amount of noise. Testing is performed on the full model, and it is somewhat mysterious that dropout works so well despite this difference between train and test. The esoteric nature of dropout has inspired a large body of work studying its regularization effects: Wager et al. (2013); Helmbold & Long (2015); Cavazza et al. (2017); Mianjy et al. (2018); Mianjy & Arora (2019) study dropout for linear models, matrix factorization, and linearized networks; Arora et al. (2020) study deep networks with dropout only at the last layer. These works primarily study simpler settings than those used in practice, and, as we demonstrate, there is an implicit regularization effect of dropout that is not adressed by prior work. A large body of recent work has studied implicit, or algorithmic regularization in deep learning, defined to be a regularization effect imposed by the training algorithm, not by the objective (see for example (Gunasekar et al., 2017; Li et al., 2017; Gunasekar et al., 2018b; Arora et al., 2019) and references therein). One notable example of this is in comparing the generalization performance of SGD vs GD: the implicit regularization effect of stochasticity in SGD has been empirically studied in the context of small v.s.