Towards Understanding the Importance of Shortcut Connections in Residual Networks

Among different types of networks, Residual Network (ResNet, He et al. (2016a)) is undoubted a milestone. ResNet is equipped with shortcut connections, which skip layers in the forward step of an input. Similar idea also appears in the Highway Networks (Srivastava et al., 2015), and further inspires densely connected convolutional networks (Huang et al., 2017). ResNet owes its great success to a surprisingly efficient training compared to the widely used feedforward Convolutional Neural Networks (CNN, Krizhevsky et al. (2012)). Feedforward CNNs are seldomly used with more than 30 layers in the existing literature. There are experimental results suggest that very deep feedforward CNNs are significantly slow to train, and yield worse performance than their shallow counterparts (He et al., 2016a). However, simple first order algorithms such as stochastic gradient descent and its variants are able to train ResNet with hundreds T. Liu, M. Chen, E. Zhou, and T. Zhao are affiliated with School of Industrial and Systems Engineering at Georgia Tech; M. Zhou is now affiliated with CS Department of Duke University; S. S. Du is now affiliated with Institute for Advanced Study; This work is done while M. Zhou is at Peking University and S. S. Du is a Ph.D. student at CMU. T. Liu and M. Chen contribute equally; T uo Zhao is the corresponding author; Email: tourzhao@gatech.edu. 1 arXiv:1909.04653v2