AutoGP: Exploring the Capabilities and Limitations of Gaussian Process Models

Recent advances in deep learning (dl; LeCun et al., 2015) have revolutionized the application of machine learning in areas such as computer vision (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012) and natural language processing (Collobert and Weston, 2008). Although certain kernel-based methods have also been successful in such domains (Cho and Saul, 2009; Mairal et al., 2014), it is still unclear whether these methods can indeed catch up with the recent dl breakthroughs. Aside from the benefits obtained from using compositional representations, we believe that the main components contributing to the success of dl techniques are: (i) their scalability to large datasets and efficient computation via gpus; (ii) their large representational power; and (iii) the use of well-targeted objective functions for the problem at hand. In the kernel world, Gaussian process (gp; Rasmussen and Williams, 2006) models are attractive because they are elegant Bayesian nonparametric approaches to learning from data. Nevertheless, besides the limitations intrinsic to local kernel machines (Bengio et al., 2005), it is clear that gp-based methods have not fully explored the desirable criteria highlighted above. Firstly, with regards to (i) scalability, despite recent advances in inducing-variable approaches and variational inference in gp models (Titsias, 2009; Hensman et al., 2013, 2015a; Dezfouli and Bonilla, 2015), the study of truly large datasets in problems other than regression and the investigation of gpu-based acceleration in gp models are still under-explored areas. We note that these issues are also shared by non-probabilistic kernel methods such as support vector machines (svms; Scholkopf and Smola, 2001). Furthermore, concerning (ii) their representational power, kernel methods have been plagued by the overuse of very limited kernels such as the squared exponential kernel, also known as the radial-basisfunction (rbf) kernel.