Statistical Guarantees and Algorithmic Convergence Issues of Variational Boosting

Variational Bayes has gained popularity in recent years as an alternative to Markov chain Monte Carlo procedures to approximate analytically intractable posterior distributions; refer to Blei et al. [2017] for a comprehensive overview. Variational inference formulates the problem of approximating the posterior as an optimization routine by minimizing a measure of discrepancy between probability densities in an approximating class and the posterior density. The variational solution refers to the closest member of the approximating class to the posterior, with closeness measured through divergences or metrics, usually Kullback-Leibler divergence. Other discrepancy measures for approximating the posterior have been studied, like the Wasserstein distance and Rényi divergence in Huggins et al. [2020], Fisher distance in Huggins et al. [2018] and Hellinger metric in Campbell and Li [2019]. The approximating class or the domain of optimization, commonly referred to as the variational family, plays a central role in these methods. It is chosen to strike a balance between computational tractability and approximation power. A richer, more flexible family allows better approximation of the posterior, while a simpler class of distributions facilitate calculations and computation speed. The Gaussian family is a popular example of a parametric variational family, where the optimization effectively takes place over a finite-dimensional parameter space. For a semi-parametric approach, one can use the popular mean-field family, which only assumes that the variational density factorizes over pre-specified sub-blocks of the parameter, with the factors otherwise unrestricted.