Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications

We investigate the concept of Best Approximation for Feedforward Neural Networks (FNN) and explore their convergence properties through the lens of Random Projection (RPNNs). RPNNs have predetermined and fixed, once and for all, internal weights and biases, offering computational efficiency. We demonstrate that there exists a choice of external weights, for any family of such RPNNs, with non-polynomial infinitely differentiable activation functions, that exhibit an exponential convergence rate when approximating any infinitely differentiable function. For illustration purposes, we test the proposed RPNN-based function approximation, with parsimoniously chosen basis functions, across five benchmark function approximation problems. Results show that RPNNs achieve comparable performance to established methods such as Legendre Polynomials, highlighting their potential for efficient and accurate function approximation. Widely used in Machine Learning (ML) tasks, including image classification, signal analysis, speech recognition, text-toimage synthesis and language processing [84, 68, 83, 37, 18, 39], the celebrated universal approximation theorem arXiv:2402.11397v1 Since the 1990s, there has been a notable increase in research focusing on solving numerical analysis problems. Lee and Kang (1990) [59] employed a Hopfield Neural Network to address the numerical solution of differential equations. Rico-Martinez et al. (1992) [76] introduced a recursive multilayer ANNs architecture, emulating the implementation of the 4th-order Runge-Kutta scheme, for the identification of continuous-time ODEs and the construction of their bifurcation diagram from experimental data. Yes ildirek and Lewis (1995) [85] employed a neural network-based controller for the feedback linearization of dynamical systems. Lagaris et al. (1998) [58] systematically investigate the use of FNNs for the solution of linear and nonlinear differential equations, addressing a range of scenarios from initial and boundary value problems. Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications and increased computational power.