Activation Learning by Local Competitions

Despite its great success, backpropagation has certain limitations that necessitate the investigation of new learning methods. In this study, we present a biologically plausible local learning rule that improves upon Hebb's well-known proposal and discovers unsupervised features by local competitions among neurons. This simple learning rule enables the creation of a forward learning paradigm called activation learning, in which the output activation (sum of the squared output) of the neural network estimates the likelihood of the input patterns, or "learn more, activate more" in simpler terms. For classification on a few small classical datasets, activation learning performs comparably to backpropagation using a fully connected network, and outperforms backpropagation when there are fewer training samples or unpredictable disturbances. Additionally, the same trained network can be used for a variety of tasks, including image generation and completion. Activation learning also achieves state-of-the-art performance on several real-world datasets for anomaly detection. This new learning paradigm, which has the potential to unify supervised, unsupervised, and semi-supervised learning and is reasonably more resistant to adversarial attacks, deserves in-depth investigation. The backpropagation algorithm [1] has driven the recent success of machine learning in tasks such as speech and image recognition [2], language processing, image and music creation, playing human games [3], etc. Many scientists argue that the backpropagation algorithm, despite being a highly effective tool for training neural networks by minimizing specific loss functions, is different from the rules governing human learning [4]-[7]. One limitation of backpropagation is that the features learned by minimizing a particular loss function tend to be task-specific. This makes it difficult for the trained models to perform generic tasks, necessitates a large amount of labeled data, and renders them vulnerable to adversarial attacks [8]. Inspired by the brain, which is believed to learn in a predominantly unsupervised fashion [9], [10], we intend to create a new learning paradigm that enables forward unsupervised training of neural networks based on a simple local learning rule while achieving comparable performance to backpropagation. The fundamental idea is that, when every neuron in a layer competes to activate while presenting distinct features, the network transmits the maximum amount of information to the next layer, and learning is enforced. Hebbian plasticity is a local correlation-based learning rule proposed by Hebb that has been supported by experimental evidences such as long-term potentiation and depression [5], [11]. It is simply phrased as'cells that fire together wire together.' H. Zhou is with the School of Information Science and Engineering, Shandong University, Qingdao, Shandong, 266237 China.