Dual Stochastic Natural Gradient Descent

Although theoretically appealing, Stochastic Natural Gradient Des cent (SNGD) [1] is computationally expensive, it has been shown to be highly sensitiv e to the learning rate, and it is not guaranteed to be convergent. Converg ent Stochastic Natural Gradient Descent (CSNGD) [6] aims at solving the last two pr oblems. However, the computational expense of CSNGD is still unacceptab le when the number of parameters is large. In this paper we introduce the Dual Stochastic Natural Gradient Descent (DSNGD) where we take benefit of dually flat manifolds to obtain a robust alternative to SNGD which is also computation ally feasible. We start by reviewing dually flat manifold concepts in section 3. Then w e introduce exponential XY families, the mathematical model required for the application of DSNGD, in section 4. After that, in section 5 we introduce DSNGD in exponential XY families under a minimal parameterization. The same idea can be extended to exponential XY families which are overparameterized.