The recently proposed stochastic Polyak stepsize (SPS) and stochastic line-search (SLS) for SGD have shown remarkable effectiveness when training over-parameterized models. However, two issues remain unsolved in this line of work.

Numerous images of atarget human subject or ascene are required for training. Second, a learned model has limited generalization capability.