Here we study structure-preserving discretizations for a certain class of dissipative (conformal) Hamiltonian systems, allowing us to analyze the symplectic structure of both Nesterov and heavy ball, besides providing several new insights into these methods. Moreover, we propose a new algorithm based on a dissipative relativistic system that normalizes the momentum and may result in more stable/faster optimization.

We want to provep is a local minimum of(11), then since (11) is a convex optimization, we can prove that p is the global optimum. We consider a closed local areaB(p,δ > 0) such that for any q B(p,δ), q 0 and we can ignore the box constraint forql for l Jc. We call a local optimal solution of(11) in B(p,δ) as p . Moreover, if kp k < 1, then we can further extendp [Jc] to produce a larger inner product withv, and this contradicts the assumption. After propagating a ballB2(µ,ρ) through a ReLU layer, we can estimate the propagated outer bound with anew ballB2(µ+,ρ)whereµ+ = max(µ,0). However, the true image ReLU(B2(µ,ρ)) has no negative elements.