nullx 0
How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGD
Neural Information Processing Systems http://nips.cc/
Continuous-time Analysis of Anchor Acceleration
Recently, the anchor acceleration, an acceleration mechanism distinct from Nes-terov's, has been discovered for minimax optimization and fixed-point problems,
Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
While our presentation focuses on this finite-sum structure, most of our convergence results can easily be adapted to the general stochastic setting (see App. D).
Nonlinear Acceleration of Stochastic Algorithms
Extrapolation methods use the last few iterates of an optimization algorithm to produce a better estimate of the optimum. They were shown to achieve optimal convergence rates in a deterministic setting using simple gradient iterates. Here, we study extrapolation methods in a stochastic setting, where the iterates are produced by either a simple or an accelerated stochastic gradient algorithm.
Supplementary Material for: Improved Algorithms for Convex-Concave Minimax Optimization 1 Some Useful Properties In this section, we review some useful properties of functions in F (m
Then, we have that 1. y Fact 2. Let z:= [ x; y ] and z This can be easily proven using the AM-GM inequality. Fact 3. Let z:= [ x; y ] R It is a crucial building block for the algorithms in this work. The following classical theorem holds for AGD. We will start by giving a precise statement of Algorithm 1.Algorithm 1 Alternating Best Response (ABR)Require: g (,), Initial point z The basic idea is the following. The following two lemmas about the inexact APP A algorithm follow from the proof of Theorem 4.1 [ Here we provide their proofs for completeness.