We demonstrate that a very deep ResNet with stacked modules that have one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in d dimensions, i.e. \ell_1(R^d). Due to the identity mapping inherent to ResNets, our network has alternating layers of dimension one and d. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension d [21,11]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.

We demonstrate that a very deep ResNet with stacked modules that have one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in d dimensions, i.e. \ell_1(R^d). Due to the identity mapping inherent to ResNets, our network has alternating layers of dimension one and d. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension d [21,11]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.

Deep neural networks are central to many recent successes of machine learning, including applications such as computer vision, natural language processing, or reinforcement learning. A common trend in deep learning has been to construct larger and deeper networks, starting from the pioneer convolutional network LeNet [19], to networks with tens of layers such as AlexNet [17] or VGG-Net [28], or recent architectures like GoogLeNet/Inception [30] or ResNet [13, 14], which may contain hundreds or thousands of layers. A typical observation is that deeper networks offer better performance. This phenomenon, at least on the training set, supports the intuition that a deeper network should have more capacity to approximate the target function, and leads to a question that has received increasing interest in the theory of deep learning: can all functions that we may care about be approximated well by a sufficiently large and deep network? In this work, we address this important question for the popular ResNet architecture.