It is a commonly held belief that enforcing invariance improves generalisation.

It is a commonly held belief that enforcing invariance improves generalisation. Although this approach enjoys widespread popularity, it is only very recently that a rigorous theoretical demonstration of this benefit has been established. In this work we build on the function space perspective of Elesedy and Zaidi arXiv:2102.10333 to derive a strictly non-zero generalisation benefit of incorporating invariance in kernel ridge regression when the target is invariant to the action of a compact group. We study invariance enforced by feature averaging and find that generalisation is governed by a notion of effective dimension that arises from the interplay between the kernel and the group. In building towards this result, we find that the action of the group induces an orthogonal decomposition of both the reproducing kernel Hilbert space and its kernel, which may be of interest in its own right.

Email: kharen@dualitygroup.com January 30, 2020 Abstract We introduce a notion of "effective dimension" of a statistical model based on the number of cubes of size 1 / n needed to cover the model space when endowed with the Fisher Information Matrix as metric, n being the number of observations. The effective dimension is then measured via the spectrum of the Fisher Information Matrix regularized using this natural scale. A very important and challenging question in statistics and machine learning is the "real" dimension of a statistical model, such as a neural network. Many definitions of effective dimension have been proposed in the literature, either based on the so-called VC dimension (see for instance [13]), or on Gardner phase-space approach [6], or also on some effective dimension based on the rank of the Jacobian matrix of the transformation between the parameters of the network and the parameters of the observable variables [2, 15] (see also [14, 1, 4, 7]). Although these notions of dimension are all very natural when the number of observations go to infinity, they do not take into account the fact that only a finite-size sample of data is available.