The Expressive Power of Neural Networks: A View from the Width

The expressive power of neural networks is important for understanding deep learning. Most existing works consider this problem from the view of the depth of a network. In this paper, we study how width affects the expressiveness of neural networks. Classical results state that depth-bounded (e.g. We show a universal approximation theorem for width-bounded ReLU networks: width-(n 4) ReLU networks, where n is the input dimension, are universal approximators.