Supplementaryfor: MomentumCenteringand Asynchronous Update for Adaptive Gradient Methods Contents

There exists an online convex optimization problem where Adam (and RMSprop) has non-zero average regret, and one of the problem is in the form ft(x)= ( Px, if t mod P =1 x, Otherwise x [ 1,1], P N,P 3 (1) Proof. See [1] Thm.1 for proof. For the problem defined above, there's a threshold of β2 above which RMSprop converge. For the problem defined by Eq. (1), ACProp algorithm converges β1,β2 (0,1), P N,P 3. Proof. We analyze the limit behavior of ACProp algorithm.