Variational Bayesian inference for linear and logistic regression

The article describe the model, derivation, and implementation of variational Bayesian inference for linear and logistic regression, both with and without automatic relevance determination. It has the dual function of acting as a tutorial for the derivation of variational Bayesian inference for simple models, as well as documenting, and providing brief examples for the MATLABfunctions that implement this inference. These functions are freely available online. 1. Introduction Linear and logistic regression are essential workhorses of statistical analysis, whose Bayesian treatment has received much recent attention (Gelman et al., 2013; Bishop, 2006; Murphy, 2012; Hastie et al., 2011). These allow specifying the a-priori uncertainty and infer a-posteriori uncertainty about regression coefficients explic-ity and hierarchically, by, for example, specifying how uncertain we are a-priori that these coefficients are small. However, Bayesian inference in such hierarchical models quickly becomes intractable, such that recent effort has focused on approximate inference, like Markov Chain Monte Carlo methods (Gilks et al., 1995), or variational Bayesian approximation (Beal, 2003; Bishop, 2006; Murphy, 2012). Here, we describe such a variational treatment and implementation of Bayesian hierarchical models for both linear and logistic regression. Even though neither the statistical models nor their Bayesian approximation are particularly novel, the article provides a tutorial-style introduction to the derivation of their algorithms, together with a MATLABimplementation of these algorithms.