On MCMC for variationally sparse Gaussian processes: A pseudo-marginal approach

Gaussian processes (GPs) are frequently used in machine learning and statistics to construct powerful models. In a Bayesian setting, GPs provide a probabilistic approach to model unknown functions; specifically, the GP prior assumes that the function evaluated at any finite set of inputs has a Gaussian distribution with consistent parameters, specified by the mean function and symmetric positive definite covariance (or kernel) function. The flexible, probabilistic and nonparametric nature of GP models makes them appropriate and useful in a wide range of applications, including geostatistics [Matheron, 1973], atmospheric sciences [Berrocal et al., 2010], biology [Stathopoulos et al., 2014], inverse problems [Kaipio and Somersalo, 2006], and more. However, when employing GPs in practice, important considerations must be made, specifically, to address the computational burden, approximation of the posterior, form of the covariance function and inference of its hyperparmeters. First, GPs suffer from a high computational burden, due to the need to store and invert large and dense covariance matrices.