Adaptive RKHS Fourier Features for Compositional Gaussian Process Models

Gaussian Processes (GPs) provide a principled Bayesian framework for function approximation, making them particularly useful in many applications requiring uncertainty calibration [Rasmussen and Williams, 2006], such as Bayesian optimisation [Snoek et al., 2012] and time-series analysis [Roberts et al., 2013]. Despite offering reasonable uncertainty estimation, shallow GPs often struggle to model complex, non-stationary processes present in practical applications. To overcome this limitation, Deep Gaussian Processes (DGPs) employ a compositional architecture by stacking multiple GP layers, thereby enhancing representational power while preserving the model's intrinsic capability to quantify uncertainty [Damianou and Lawrence, 2013]. However, the conventional variational formulation of DGPs heavily depends on local inducing point approximations across intermediate GP layers [Titsias, 2009, Salimbeni and Deisenroth, 2017], which hinder the model from capturing the global structures commonly found in real-world scenarios. Incorporating Fourier features into GP models has shown promise in addressing this challenge in GP inference due to the periodic nature of these features. A line of research uses Random Fourier Features (RFFs, [Rahimi and Recht, 2007]) of stationary kernels to convert the original (deep) GPs into Bayesian networks in weight space [Lázaro-Gredilla et al., 2010, Gal and Turner, 2015, Cutajar et al., 2017]. Building on this concept within a sparse variational GP framework, recent advancements in inter-domain GPs [Lázaro-Gredilla and Figueiras-Vidal, 2009a, Van der Wilk et al., 2020] directly approximate the posterior of the original GPs by introducing fixed Variational Fourier Features (VFFs) through process projection onto a Reproducing Kernel Hilbert Space (RKHS)[Hensman et al., 2018, Rudner et al., 2020]. VFFs are derived by projecting GPs onto a different domain.