ResNet with one-neuron hidden layers is a Universal Approximator

Deep neural networks are central to many recent successes of machine learning, including applications such as computer vision, natural language processing, or reinforcement learning. A common trend in deep learning has been to construct larger and deeper networks, starting from the pioneer convolutional network LeNet [19], to networks with tens of layers such as AlexNet [17] or VGG-Net [28], or recent architectures like GoogLeNet/Inception [30] or ResNet [13, 14], which may contain hundreds or thousands of layers. A typical observation is that deeper networks offer better performance. This phenomenon, at least on the training set, supports the intuition that a deeper network should have more capacity to approximate the target function, and leads to a question that has received increasing interest in the theory of deep learning: can all functions that we may care about be approximated well by a sufficiently large and deep network? In this work, we address this important question for the popular ResNet architecture.