Gaussian process interpolation: the choice of the family of models is more important than that of the selection criterion

Regression and interpolation with Gaussian processes, or kriging, is a popular statistical tool for non-parametric function estimation, originating from geostatistics and time series analysis, and later adopted in many other areas such as machine learning and the design and analysis of computer experiments (see, e.g., Stein, 1999; Santner et al., 2003; Rasmussen and Williams, 2006, and references therein). It is widely used for constructing fast approximations of time-consuming computer models, with applications to calibration and validation (Kennedy and O'Hagan, 2001; Bayarri et al., 2007), engineering design (Jones et al., 1998; Forrester et al., 2008), Bayesian inference (Calderhead et al., 2009; Wilkinson, 2014), and the optimization of machine learning algorithms (Bergstra et al., 2011)--to name but a few. A Gaussian process (GP) prior is characterized by its mean and covariance functions. They are usually chosen within parametric families (for instance, constant or linear mean functions, and Matérn covariance functions), which transfers the problem of choosing the mean and covariance functions to that of selecting parameters. The selection is most often carried out by optimization of a criterion that measures the goodness of fit of the predictive distributions, and a variety of such criteria--the likelihood function, the leave-one-out (LOO) squared-predictionerror criterion (hereafter denoted by LOO-SPE), and others--is available from the literature.