[P] Going Deeper: Infinite Deep Neural Networks • r/MachineLearning

The described "meta-layer" with infinite many sub-layers has a ResNet-like structure (see Figure 3 on page 5 on the linked github page). This means every sub-layer has a structure like f(x) x g(x). Every new added layer, therefore, has this structure. To make the process of adding layers smoother there is a factor "d" between 0 and 1 added to this layer-defintion: f(x) x d*g(x). Usually "adding" (more correct: activating; because the model assumes that there are from the beginning infinite many layers) a new layer requires multiple iterations where "d" grows from 0 up to a value smaller than 1 (the limit is 1). Therefore, if a new layer is added, the value of "d" is very small, e.g.