A scale-dependent notion of effective dimension

Email: kharen@dualitygroup.com January 30, 2020 Abstract We introduce a notion of "effective dimension" of a statistical model based on the number of cubes of size 1 / n needed to cover the model space when endowed with the Fisher Information Matrix as metric, n being the number of observations. The effective dimension is then measured via the spectrum of the Fisher Information Matrix regularized using this natural scale. A very important and challenging question in statistics and machine learning is the "real" dimension of a statistical model, such as a neural network. Many definitions of effective dimension have been proposed in the literature, either based on the so-called VC dimension (see for instance [13]), or on Gardner phase-space approach [6], or also on some effective dimension based on the rank of the Jacobian matrix of the transformation between the parameters of the network and the parameters of the observable variables [2, 15] (see also [14, 1, 4, 7]). Although these notions of dimension are all very natural when the number of observations go to infinity, they do not take into account the fact that only a finite-size sample of data is available.