On an Interpretation of ResNets via Solution Constructions

He et al. (2016a) introduced a type of shortcut connection in the architecture of a feedforward neural network, which has been proved effective in the learning of particularly deep neural networks. The modified architecture is called residual network (ResNet), which is widely applied and nearly becomes a standard component of network architectures, such as in Transformer (Vaswani et al., 2017). Note that in He et al. (2016b), another proposed shortcut connection is slightly different from that of He et al. (2016a) in whether or not a ReLU is used after an addition operation. Both of the above two versions are called ResNet and this paper will study the former one.