Limitations on approximation by deep and shallow neural networks

Since neural network approximation is the method of choice in building numerical algorithms in many application areas, it is important to understand not only how well they approximate but also any lower bounds on their approximation power. In this paper, we study the limitations of deep and shallow neural networks to approximate a compact subset K X of a Banach space X when it is required that the parameters in the approximation procedure have certain bounds. This is done by proving appropriate Carl's type inequalities that relate the error of neural network approximation of K to the entropy numbers of this set. We consider feed-forward neural networks (NN) with ReLU or Lipschitz sigmoidal activation functions, width W 2 and depth n, whose parameters have absolute values bounded by a given function w ( n). We prove that the capabilities of these networks to approximate any compact subset K is limited by the behavior of its entropy numbers.