Reviews: The Physical Systems Behind Optimization Algorithms

The paper presents a continuous-time ODE interpretation of four popular optimization algorithms: Gradient descent, proximal gradient descent, coordinate gradient decent and Newton's method. The four algorithms are all interpreted as damped oscillators with different mass and damping coefficients. It is shown that this ODE formulation can be used to derive the (known) convergence rates in a fairly straight forward manner. Further, the ODE formulation allows to analyze convergence in the non-convex case under the PL-condition. An extension to nonsmooth composite optimization is also discussed.