s-tllr
TESS: A Scalable Temporally and Spatially Local Learning Rule for Spiking Neural Networks
Apolinario, Marco Paul E., Roy, Kaushik, Frenkel, Charlotte
The demand for low-power inference and training of deep neural networks (DNNs) on edge devices has intensified the need for algorithms that are both scalable and energy-efficient. While spiking neural networks (SNNs) allow for efficient inference by processing complex spatio-temporal dynamics in an event-driven fashion, training them on resource-constrained devices remains challenging due to the high computational and memory demands of conventional error backpropagation (BP)-based approaches. In this work, we draw inspiration from biological mechanisms such as eligibility traces, spike-timing-dependent plasticity, and neural activity synchronization to introduce TESS, a temporally and spatially local learning rule for training SNNs. Our approach addresses both temporal and spatial credit assignments by relying solely on locally available signals within each neuron, thereby allowing computational and memory overheads to scale linearly with the number of neurons, independently of the number of time steps. Despite relying on local mechanisms, we demonstrate performance comparable to the backpropagation through time (BPTT) algorithm, within $\sim1.4$ accuracy points on challenging computer vision scenarios relevant at the edge, such as the IBM DVS Gesture dataset, CIFAR10-DVS, and temporal versions of CIFAR10, and CIFAR100. Being able to produce comparable performance to BPTT while keeping low time and memory complexity, TESS enables efficient and scalable on-device learning at the edge.
S-TLLR: STDP-inspired Temporal Local Learning Rule for Spiking Neural Networks
Apolinario, Marco Paul E., Roy, Kaushik
Spiking Neural Networks (SNNs) are biologically plausible models that have been identified as potentially apt for deploying energy-efficient intelligence at the edge, particularly for sequential learning tasks. However, training of SNNs poses significant challenges due to the necessity for precise temporal and spatial credit assignment. Back-propagation through time (BPTT) algorithm, whilst the most widely used method for addressing these issues, incurs a high computational cost due to its temporal dependency. In this work, we propose S-TLLR, a novel threefactor temporal local learning rule inspired by the Spike-Timing Dependent Plasticity (STDP) mechanism, aimed at training deep SNNs on event-based learning tasks. Furthermore, S-TLLR is designed to have low memory and time complexities, which are independent of the number of time steps, rendering it suitable for online learning on low-power edge devices. To demonstrate the scalability of our proposed method, we have conducted extensive evaluations on event-based datasets spanning a wide range of applications, such as image and gesture recognition, audio classification, and optical flow estimation. In all the experiments, S-TLLR achieved high accuracy, comparable to BPTT, with a reduction in memory between 5 50 and multiply-accumulate (MAC) operations between 1.3 6.6 . Over the past decade, the field of artificial intelligence has undergone a remarkable transformation, driven by a prevalent trend of continuously increasing the size and complexity of neural network models. While this approach has yielded remarkable advancements in various cognitive tasks (Brown et al., 2020; Dosovitskiy et al., 2021), it has come at a significant cost: AI systems now demand substantial energy and computational resources. This inherent drawback becomes increasingly apparent when comparing the energy efficiency of current AI systems with the remarkable efficiency exhibited by the human brain (Roy et al., 2019; Gerstner et al., 2014; Christensen et al., 2022; Eshraghian et al., 2023).