Delving Deeper Into Astromorphic Transformers

--Preliminary attempts at incorporating the critical role of astrocytes--cells that constitute more than 50% of human brain cells--in brain-inspired neuromorphic computing remain in infancy. This paper seeks to delve deeper into various key aspects of neuron-synapse-astrocyte interactions to mimic self-attention mechanisms in Transformers. The cross-layer perspective explored in this work involves bioplausible modeling of Hebbian and presynaptic plasticities in neuron-astrocyte networks, incorporating effects of non-linearities and feedback along with algorithmic formulations to map the neuron-astrocyte computations to self-attention mechanism and evaluating the impact of incorporating bio-realistic effects from the machine learning application side. Our analysis on sentiment and image classification tasks (IMDB and CIF AR10 datasets) highlights the advantages of Astromorphic Transformers, offering improved accuracy and learning speed. Furthermore, the model demonstrates strong natural language generation capabilities on the WikiT ext-2 dataset, achieving better perplexity compared to conventional models, thus showcasing enhanced generalization and stability across diverse machine learning tasks. STROCYTES, a type of glial cell, play a critical role in brain function, encompassing various processes such as homeostasis, metabolism, and synaptic regulation [1]. Astrocytes detect and regulate synaptic activity in the tripartite synapse through interactions with pre-and postsynaptic neurons. Investigating their impact on neural computation is currently an active research field in neuroscience and underscores the critical need to move beyond the neuro-synaptic perspective of current Artificial Intelligence (AI) systems. Recent experimental findings on neuron-astrocyte interactions and modulation have led to significant progress in computational neuroscience, enabling the development of models that incorporate neuron-astrocyte interactions within neural networks [2], [3]. Astrocytes have been found to modulate bursting in neural circuitry through the release of gliotransmitters, which have an impact on neuronal excitability and synaptic plasticity [4], [5]. Astrocytes possess the ability to encode information through calcium signaling and regulate information processing, thereby actively engaging in neural computation at the tripartite synapse level. Additionally, astrocytes possess inherent capacity as memory components [6], [7] and plasticity regulators that are capable of facilitating local sequential learning [8], [9].