Stationary Activations for Uncertainty Calibration in Deep Learning

We introduce a new family of non-linear neural network activation functions that mimic the properties induced by the widely-used Mat\'ern family of kernels in Gaussian process (GP) models.