trish
A Stochastic Trust Region Algorithm
Curtis, Frank E., Scheinberg, Katya, Shi, Rui
The stochastic gradient (SG) method is the signature strategy for solving stochastic and finite-sum minimization problems. In this iterative approach, each step to update the solution estimate is obtained by taking a negative multiple of an unbiased gradient estimate. With careful choices for the stepsize sequence, the SG method possesses convergence guarantees and has been employed to great success for solving various types of problems, such as those arising in machine learning. One disadvantage of the SG method is that stochastic gradients, like the gradients that they approximate, possess no natural scaling. By this, we mean that in order to guarantee convergence, the algorithm needs to choose stepsizes in a problem-dependent manner; e.g., common theoretical guarantees require that the stepsize is proportional to 1/L, where L is a Lipschitz constant for the gradient of the objective function.