We investigate random feature models in which neural networks sampled from a prescribed initialization ensemble are frozen and used as random features, with only the readout weights optimized. Adopting a statistical-physics viewpoint, we study the training, test, and generalization errors beyond the mean-kernel approximation. Since the predictor is a nonlinear functional of the induced random kernel, the ensemble-averaged errors depend not only on the mean kernel but also on higher-order fluctuation statistics. Within an effective field-theoretic framework, these finite-width contributions naturally appear as loop corrections. We derive the loop corrections to the training, test, and generalization errors, obtain their scaling laws, and support the theory with experimental verification.

But how to explain this phenomenon has been largely ignored by existing studies.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

We theoretically study the landscape of the training error for neural networks in overparameterized cases.

Our first algorithm consists of a simple modification of the ML model used by Lewis et al. and applies to a property of interest known in advance.

This important phenomenon commonly appears in implemented neural network architectures, and also seems to emerge in epoch-wise curves during the training process.