Gradient Descent
Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time
They do not study how the population loss decreases during the first stage.
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Machine unlearning has raised significant interest with the adoption of laws ensuring the "right to be forgotten".
An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness
However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient).
Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates
In particular, we establish the surprising result that: F or any constant learning rate ฮท > 0, the stochastic gradient bandit algorithm is guaranteed to converge to the globally optimal policy almost surely.
Behavior Alignment via Reward Function Optimization Dhawal Gupta University of Massachusetts Y ash Chandak
Designing reward functions for efficiently guiding reinforcement learning (RL) agents toward specific behaviors is a complex task.
Stable Nonconvex-Nonconcave Training via Linear Interpolation
By replacing the inner optimizer in RAPP we rediscover the family of Lookahead algorithms for which we establish convergence in cohypomonotone problems even when the base optimizer is taken to be gradient descent ascent.
Online Performative Gradient Descent for Learning Nash Equilibria in Decision-Dependent Games Zihan Zhu Duke University Ethan X. Fang Duke University Zhuoran Yang Yale University
We focus on finding the Nash equilibrium of decision-dependent games in the bandit feedback setting.
The Sample Complexity of Gradient Descent in Stochastic Convex Optimization Roi Livni School of Electrical Engineering Tel Aviv University rlivni@tauex.tau.ac.il
But, also, it has become focus of study because it is one of few theoretical settings that exhibit overparameterized learning .
Transformers learn to implement preconditioned gradient descent for in-context learning
Several recent works demonstrate that transformers can implement algorithms like gradient descent. By a careful construction of weights, these works show that multiple layers of transformers are expressive enough to simulate iterations of gradient descent.
Shuffling Gradient-Based Methods for Nonconvex-Concave Minimax Optimization
It also employs a new shuffling estimator for the "hyper-gradient",