For both learning rules, we prove overfitting is tempered.

Wefindalarge range of behavior that can be precisely characterized by a new measure ofconfounding strength.

Currently, most theoretical works on implicit regularisation have primarily focused on continuous time approximations of (S)GD where the impact of crucial hyperparameters such as the stepsize and the minibatch size are ignored. One such common simplification is to analyse gradient flow, which is a continuous time limit of GD and minibatch SGD with an infinitesimal stepsize. By definition, this analysis does not capture the effect of stepsize or stochasticity.

The next lemma was used in the proof of Theorem 1.

And, conversely, the minimum-norm interpolator is the solution to adversarial training with a given radius.