Reviews: Reducing Reparameterization Gradient Variance

Summary This paper proposes a control variate (CV) for the reparametrization gradient by exploiting a linearization of the data model score. For Gaussian random variables, such a linearization has a distribution with a known mean, allowing its use as a CV. Experiments show using the CV results in faster (according to wall clock time) ELBO optimization for a GLM and Bayesian NN. Furthermore, the paper reports 100 fold () variance decreases during optimization of the GLM. Evaluation Method: The CV proposed is clever; the observation that the linearization of the data score has a known distribution is non-obvious and interesting. This is a contribution that can easily be incorporated when using the reparametrization trick.