Kernel Regression with Infinite-Width Neural Networks on Millions of Examples

Kernel methods are often contrasted with deep learning, but recent advances in machine learning have identified and developed exciting correspondences [Lee et al., 2018, Matthews et al., 2018, Jacot et al., 2018]. While a useful method in its own right, kernel regression has been used to better understand neural networks and deep learning. More specifically, if the parameters of a neural network are treated as random variables whose distribution is set by the initialization, we can view the neural network as a random function. Then as the width of the network becomes large, the distribution of this random function is a Gaussian process with a specific covariance function or kernel. We refer to kernels that arise from this connection with infinite-width neural networks as neural kernels. The specific kernel is determined by the architecture, inference type, and other hyperparameters of the neural network. Moreover, the connection between neural networks and Gaussian processes has generated many high-performance kernels for diverse or nonstandard data modalities, such as images, sequences, and graphs. This performance often comes at a cost, as the kernels require significantly more compute than standard kernels such as RBFs.