Precise analysis of ridge interpolators under heavy correlations -- a Random Duality Theory view

While non-monotonic behavior in parametric characterizations of various random structures has been known for a long time, it has received an enormous amount of attention in recent years. By a no surprise, a larger than ever popularity of machine learning (ML) and neural networks (NN) substantially contributed to this. Two lines of work, the neural network one (see, e.g., [5, 7, 8, 41, 60, 61]) and the statistical one (see, e.g., [6,16,17,24]), together with their interconnections, are among those that, in our view, led the way in bringing such a strong interest to these phenomena. The first line, (in a way (re)initiated in [5, 7, 8, 41, 60, 61] (and further theoretically substantiated in e.g., [6, 36])), relates to the empirical observation of the so-called double-descent phenomenon in neural networks generalization abilities. In particular, it was emphasized in [5] that the generalization error has a U-shape dependence on the size of the network before and a (potentially surprising) second descent after the interpolating limit (see, also [1, 36, 37, 60, 61] and particularly [56] for early double-decent displays).