Ultra-Wide Deep Nets and the Neural Tangent Kernel (NTK)

Illustration by Belkin et al. (2018) of the effect of increased model complexity on generalization: traditional belief (a) vs actual practice (b). Traditional wisdom in machine learning holds that there is a careful trade-off between training error and generalization gap. There is a "sweet spot" for the model complexity such that the model (i) is big enough to achieve reasonably good training error, and (ii) is small enough so that the generalization gap – the difference between test error and training error – can be controlled. A smaller model would give a larger training error while making the model bigger would result in a larger generalization gap, both leading to larger test errors. This is described by the classical U-shaped curve for the test error when the model complexity varies (see Figure 1(a)).