Information Geometry of Dropout Training

Deep neural networks have been experimentally successful in a variety of fields (Deng and Yu, 2014; LeCun et al., 2015; Goodfellow et al., 2016). Dropout is one of the techniques that contribute to the performance improvement of neural networks (Srivastava et al., 2014). Many experimental results have reported the effectiveness of dropout, making it an important technique for training neural networks (Wu and Gu, 2015; Pham et al., 2014; Park and Kwak, 2016; Labach et al., 2019). Furthermore, the simplicity of the idea of dropout has led to the proposal of a great number of variants (Iosifidis et al., 2015; Moon et al., 2015; Gal et al., 2017; Zolna et al., 2017; Hou and Wang, 2019; Keshari et al., 2019; Ma et al., 2020). Understanding the behavior of such an important technique can be a way to know which of these variants to use, and in what cases dropout is effective in the first place.