Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference

Understanding the gradient variance of blackbox Despite the advances of BBVI, little is known about its theoretical variational inference (BBVI) is a crucial step properties. Even when restricted to the locationscale for establishing its convergence and developing family (Definition 2), it is unknown whether BBVI algorithmic improvements. However, existing is guaranteed to converge without having to modify the studies have yet to show that the gradient variance algorithms used in practice, for example, by enforcing of BBVI satisfies the conditions used to bounded domains, bounded support, bounded gradients, study the convergence of stochastic gradient descent and such. This theoretical insight is necessary since BBVI (SGD), the workhorse of BBVI. In this methods are known to be less robust (Yao et al., 2018; work, we show that BBVI satisfies a matching Dhaka et al., 2020; Welandawe et al., 2022; Dhaka et al., bound corresponding to the condition used 2021; Domke, 2020) compared to other inference methods in the SGD literature when applied to smooth and such as Markov chain Monte Carlo. Although progress has quadratically-growing log-likelihoods. Our results been made to formalize the theory of BBVI with some generality, generalize to nonlinear covariance parameterizations the gap between our understanding of BBVI and the widely used in the practice of BBVI.