Gaussian Process Boosting

In this article, we propose a novel way to combine boosting with Gaussian process and mixed effects models. Boosting [Freund and Schapire, 1996, Breiman, 1998, Friedman et al., 2000, Mason et al., 2000, Friedman, 2001, Bühlmann and Hothorn, 2007] is a machine learning technique that achieves superior predictive performance for a large variety of datasets [Chen and Guestrin, 2016, Nielsen, 2016]. Apart from this, the wide adoption of treeboosting in applied machine learning and data science is due to several advantages: boosting with trees as base learners can automatically account for complex non-linearities, discontinuities, and high-order interactions, it is robust to outliers in and multicollinearity among predictor variables, it is scale-invariant to monotone transformations of the predictor variables, it can handle missing values in predictor variables automatically by loosing almost no information [Elith et al., 2008], and boosting can perform variable selection. Gaussian processes [Williams and Rasmussen, 2006], on the other hand, are flexible nonparametric function models that achieve state-of-the-art predictive accuracy and allow for making probabilistic predictions [Gneiting et al., 2007]. Gaussian process and mixed effects models are used, for instance, for nonparametric regression, modeling of time series [Shumway and Stoffer, 2017], spatial [Banerjee et al., 2014], spatiotemporal [Cressie and Wikle, 2015], panel or longitudinal, and hierarchically clustered or grouped