Asymptotic Consistency of Loss-Calibrated Variational Bayes

Consider a loss function G ( a,θ) ( a,θ) G ( a,θ) R, where a A R s is a decision/design variable and θ Θ R d is a model parameter space. Given a set of observations X n {ξ 1,...,ξ n} drawn from a distribution with unknown parameter θ 0, p( X n θ 0), our goal is to compute the Bayes optimal decision rule a ( X n) arg min a A E π[G ( a,θ)] ΘG ( a,θ) π ( θ X n) dθ, (1) where π ( θ X n) is the posterior distribution. The latter results when a Bayesian decision-maker places a prior distribution π ( θ) over the parameter space Θ, capturing a priori information about θ such as location or spread. Given X n, the prior and likelihood p ( X n θ) together define a posterior distribution π ( θ X n) p ( X n θ) π ( θ) p( θ, X n), the conditional distribution over θ given observations. The posterior distribution represents uncertainty over the unknown parameter θ, and contains all information required for further inferences or optimization. In general, under most realistic modeling assumptions, closed-form analytic expressions are unavailable for π ( θ X n), making the subsequent integration and optimization problems intractable. In practice, therefore, one uses an approximation to the posterior in the integration in (1). It is easy to see that posterior computation can be expressed as a convex optimization problem: min q () M KL( q ( θ) π ( θ X n)) KL( q ( θ) p ( θ, X n)) log p( X n) (2) KL( q ( θ) π ( θ)) Θlog p( X n θ) q ( θ) dθ log p ( X n) where KL is the Kullback-Leibler divergence and M is the space of all distributions that are absolutely continuous with respect to the posterior (or, equivalently, the prior).