T,i. Without further assumption, Pd i=1s 1/2

It's related to "cycle" in theory, and "mode collapse" in practice.(e)28 We refer to all three optimizers. Fig2 is illustrative; rigorously, oscillation amplitude29 in y-axis decreases, but gradient is independent of the distance to axis for L1 loss, hence our analysis holds for30 both fixed-step-size and decreasing-step-size. (g) We absorbinto st in theoretical analysis, in implementation31 weadd tomatchassumptionst >c>0inTheorem2.1(c It might be better to directly approximateH 1 f rather than approximating H as diag(H). Comparison with Adam We address R6's concern that the success of AdaBelief stems largely from an41 effectively larger stepsize.