Provable wavelet-based neural approximation

Provable wavelet-based neural approximation Youngmi Hur Hyojae Lim Mikyoung Lim April 24, 2025 Abstract In this paper, we develop a wavelet-based theoretical framework for analyzing the universal approximation capabilities of neural networks over a wide range of activation functions. Leveraging wavelet frame theory on the spaces of homogeneous type, we derive sufficient conditions on activation functions to ensure that the associated neural network approximates any functions in the given space, along with an error estimate. These sufficient conditions accommodate a variety of smooth activation functions, including those that exhibit oscillatory behavior. Furthermore, by considering the L 2 -distance between smooth and non-smooth activation functions, we establish a generalized approximation result that is applicable to non-smooth activations, with the error explicitly controlled by this distance. This provides increased flexibility in the design of network architectures. 1 Introduction Neural networks have long been recognized for their remarkable ability to approximate a wide range of functions, enabling state-of-the-art achievements across various fields in machine learning and artificial intelligence, image processing, natural language processing, and scientific computing (see, for example, [13, 19] and references therein). Various activation functions, such as ReLU, Sigmoid, Tanh, and oscillatory functions, have also been explored to further enhance network performance and adaptability. The versatility of neural networks originates from the structural flexibility of architectures that combine affine transformations with nonlinear activation functions. In addition, classical universal approximation theorems [5, 12, 16] provide a theoretical basis for this flexibility by guaranteeing that, under suitable conditions, neural networks can approximate any continuous function on a bounded domain, underscoring their representational power. These seminal results have been extended along various directions, including radial basis function (RBF) networks [22, 25], non-polynomial activations [20], approximation of functions and their derivatives [15, 21], the influence of network depth [9], approximation error bounds [1], convolutional neural networks (CNN) [32], recurrent neural networks (RNN) [27]. As neural network architectures continue to evolve and diversify in practice, their theoretical foundations-beyond those provided by classical approximation theorems-have attracted Department of Mathematics, Yonsei University, Seoul 03722, Republic of Korea (yhur@yonsei.ac.kr)