On the Role of Optimization in Double Descent: A Least Squares Study

One of their most intriguing features is their ability to perform better with scale. Indeed, some of the most impressive results [see e.g.