Lazy learning: a biologically-inspired plasticity rule for fast and energy efficient synaptic plasticity

When training neural networks for classification tasks with backpropagation, parameters are updated on every trial, even if the sample is classified correctly. In contrast, humans concentrate their learning effort on errors. Inspired by human learning, we introduce lazy learning, which only learns on incorrect samples. Lazy learning can be implemented in a few lines of code and requires no hyperparameter tuning. Lazy learning achieves state-of-the-art performance and is particularly suited when datasets are large. For instance, it reaches 99.2% test accuracy on Extended MNIST using a single-layer MLP, and does so 7.6 faster than a matched backprop network. Recent progress in machine learning has been partly attributed to the use of large data sets [LeCun et al., 2015]. Even already large datasets are often augmented to further boost performance. However, repeatedly cycling over large datasets and adjusting the parameters is time and energy consuming. In classification tasks, backprop typically prescribes synaptic updates regardless of whether the classification was correct or incorrect; updating the network to be correct if it was wrong, but also updating to be more correct if it was right.