We consider the problem of regret minimization in non-parametric stochastic bandits. When the rewards are known to be bounded from above, there exists asymptotically optimal algorithms, with asymptotic regret depending on an infi-mum of Kullback-Leibler divergences (KL).

We consider the problem of fitting variational posterior approximations using stochastic optimization methods.

We demonstrate that, surprisingly, the expected value ofthegradient isnotalwaysthedirection maximizing theprobability ofdescent, and in fact, these directions may be nearly orthogonal. This observation then inspires an elegant optimization scheme seeking to maximize the probability of descent while moving in the direction of most-probable descent.