RFFNet: Scalable and interpretable kernel methods via Random Fourier Features

Kernel methods provide a flexible and theoretically grounded approach to nonlinear and nonparametric learning. While memory requirements hinder their applicability to large datasets, many approximate solvers were recently developed for scaling up kernel methods, such as random Fourier features. However, these scalable approaches are based on approximations of isotropic kernels, which are incapable of removing the influence of possibly irrelevant features. In this work, we design random Fourier features for automatic relevance determination kernels, widely used for variable selection, and propose a new method based on joint optimization of the kernel machine parameters and the kernel relevances. Additionally, we present a new optimization algorithm that efficiently tackles the resulting objective function, which is non-convex. Numerical validation on synthetic and real-world data shows that our approach achieves low prediction error and effectively identifies relevant predictors. Our solution is modular and uses the PyTorch framework.