Stochastic Gradient Descent (SGD) has been the method of choice for learning large-scale non-convex models. While a general analysis of when SGD works has been elusive, there has been a lot of recent progress in understanding the convergence of Gradient Flow (GF) on the population loss, partly due to the simplicity thatacontinuous-time analysis buysus.

Meta learning aims to learn the inductive biases ( e .

However,sincetheinitial large learning rate generally helps the optimizer to converge to flatter minima, we hypothesize that the winning tickets have relatively sharp minima, which is considered a disadvantage in terms of generalization ability.

Theproblem is yet challenging and traditionally requires manual efforts from experts ofprofessional artists and a largeamount oftimecost.

The most extensive model described to date is the base model size encompasses 77.6M parameters

Itfollows from (2) thatX Np(0, (eB,e )), where (eB,e ): = (I eB) Te (I eB) 1. (3) Wewillassumethat 0, andmoreoverthatrmin( ) rmax( ) 1, i.e. theeigenvaluesof are boundedawayfrom0and1. See (47) inthesupplement ( ;s), which conditionnumber ofsizeO(s).