[R] Hamiltonian Descent Methods • r/MachineLearning

TL,DR: We're not sure yet what this means for a neural net user. But, there is a connection to RMSProp and Adam, which we explore in the preprint (see the relativistic kinetic energy subsection). Given that the kinetic energy is a design choice, there's lots to explore. Keep in mind that these methods are generalizations of the momentum method, which is one of the more popular neural network optimizers. In the preprint we very briefly consider conditions that guarantee convergence to stationary points for non-convex (i.e., neural nets) functions.