Risk Bounds for Multi-layer Perceptrons through Spectra of Integral Operators

Multi-layer perceptrons have been an important family of machine learning methods, whose history alternates between periods of wide popularity and periods of fading interest [21, 18, 11]. Described in a mathematical language, a multi-layer perceptron is the iterated composition of parameterized affine maps and nonlinear maps, ultimately composed with a prediction map either for the purpose of predicting a scalar value as in curve fitting or regression, or a binary label as in supervised classification. As the number of layers or network depth grows, the parameterized map resulting from this iterated composition may change in terms of smoothness properties. However the impact of network depth on the regularity of the resulting deep network function is, quite surprisingly, still not fully understood, in particular for multi-layer perceptrons. Consider the setting, where the number of datapoints is finite, the number of hidden layers is finite, and the number of weights per layers is finite.