Spikes as regularizers

The perceptron [Rosenblatt, 1958] is a conceptually simple and widely used discriminative and linear classification algorithm. It was originally motivated by observations of how signals are passed between neurons in the brain. We will return to the perceptron as a model of neural computation, but from a more technical point of view, the main weakness of the perceptron as a linear classifier is that it is prone to overfitting. One particular type of overfitting that is likely to happen in perceptron learning is feature swamping [Sutton et al., 2006], i.e., that very frequent features may prevent covariant features from being updated, leading to catastrophic performance if the frequent features are absent or less frequent at test time. In other words, in the perceptron, as well as in passive-aggressive learning Crammer et al. [2006], parameters are only updated when features occur, and rare features therefore often receive inaccurate values. There are several ways to approach such overfitting, e.g., capping the model's supremum norm, but here we focus on a specific line of research: confidence-weighted learning of linear classifiers. Confidence-weighted learning explicitly estimates confidence during induction, often by maintaining Gaussian distributions over parameter vectors. In other words, each model parameter is interpreted as a mean, and augmented with a covariance estimate. Confidence-Weighted Learning CWL [Dredze et al., 2008] was the first learning algorithm to do this, but Crammer et al. [2009] later introduced Adaptive Regularization of Weight Vectors (A