The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Losses

Peking University Abstract: The backpropagation algorithm, despite its widespread use in neural network learning, may not accurately emulate the human cortex's learning process. However, the original FFA paper and related works on the Forward-Forward Algorithm only mentioned very limited types of neural network mechanisms and may limit its application and effectiveness. In response to these challenges, we propose an integrated method that combines the strengths of both FFA and shallow backpropagation, yielding a biologically plausible neural network training algorithm which can also be applied to various network structures. We applied this integrated approach to the classification of the Modified National Institute of Standards and Technology (MNIST) database, where it outperformed FFA and demonstrated superior resilience to noise compared to backpropagation. We show that training neural networks with the Integrated Forward-Forward Algorithm has the potential of generating neural networks with advantageous features like robustness. 1. Introduction For the past decade, Deep learning has made significant progress on countless tasks and problems, with backpropagation as a key contributing factor to train the deep models developed. However, in aspects of biological plausibility, although the model of deep neural networks for artificial intelligence is initially inspired by biological neurons' structures, the training process of these deep neural networks using backpropagation throughout the whole neural network does not seem biologically plausible. Researchers have argued that the long range, or global, backpropagation of gradient information has little evidence in neuron science and violates many principles of the cortex learning found in neural science, like locality and onlinety. The original Forward-Forward Algorithm is one of the approaches in biological plausible learning as an alternative of backpropagation to train neural networks initially proposed by Hinton(2022) [1].