Growing axons: greedy learning of neural networks with application to function approximation

Deep neural networks (DNN) have achieved tremendous success in many areas, including image processing, natural language processing, video, and audio synthesis. They have also been used for a long time as a general tool for solving regression tasks, i.e., an approximation of a given function from its samples. Neural networks are known to be a universal approximator for continuous functions [6, 3]. Recently, several approximation rate results have been established: it has been shown that a certain class of deep neural networks with ReLU [11] activation functions provide guaranteed convergence rates for certain function classes [16, 5, 9]. Recent paper [12] provides expressive power results for general piecewise analytic functions with point singularities.