Gradient Descent
Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, Pascal Vincent
Stochastic Gradient Descent (SGD) and its variants are the current workhorse for training neural networks.
Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
Marton Havasi, José Miguel Hernández-Lobato, Juan José Murillo-Fuentes
Neural Information Processing Systems http://nips.cc/
Zeroth-order (Non)-Convex Stochastic Optimization via Conditional Gradient and Gradient Updates
Krishnakumar Balasubramanian, Saeed Ghadimi
Neural Information Processing Systems http://nips.cc/
ATOMO: Communication-efficient Learning via Atomic Sparsification
Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, Stephen Wright
Neural Information Processing Systems http://nips.cc/
The promises and pitfalls of Stochastic Gradient Langevin Dynamics
Nicolas Brosse, Alain Durmus, Eric Moulines
Stochastic Gradient Langevin Dynamics (SGLD) has emerged as a key MCMC algorithm for Bayesian learning from large scale datasets.
Gradient Sparsification for Communication-Efficient Distributed Optimization
Jianqiao Wangni, Jialei Wang, Ji Liu, Tong Zhang
In the synchronous stochastic gradient method, each worker processes a random minibatch of its training data, and then the local updates are synchronized by making an All-Reduce step, which aggregates stochastic gradients from all workers, and taking a Broadcast step that transmits the updated parameter vector back to all workers.
cpSGD: Communication-efficient and differentially-private distributed SGD
Naman Agarwal, Ananda Theertha Suresh, Felix Xinnan X. Yu, Sanjiv Kumar, Brendan McMahan
Distributed stochastic gradient descent is an important subroutine in distributed learning.
Gradient Descent for Spiking Neural Networks
Dongsung Huh, Terrence J. Sejnowski
Research in spike-based computation has been impeded by the lack of efficient supervised learning algorithm for spiking neural networks.
SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path-Integrated Differential Estimator
Cong Fang, Chris Junchi Li, Zhouchen Lin, Tong Zhang
Neural Information Processing Systems http://nips.cc/
The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization
Constantinos Daskalakis, Ioannis Panageas
When they converge, do they converge to local min-max solutions? We characterize the limit points of two basic first order methods, namely Gradient Descent/Ascent (GDA) and Optimistic Gradient Descent Ascent (OGDA).