Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems

One of the key goals of brain-inspired computing is to develop methods that draw inspiration from computational neuroscience and cognitive science to build effective adaptive and efficient agents that are capable of intelligently interacting with their environment. Notably, brain-inspired computational research seeks to develop intelligent systems that are capable of circumventing the current limitations of modern-day approaches [1, 2], such as deep neural networks trained by the popular backpropagation of errors (or backprop)[3]. This goal is complementary to (and, to an extent, even a precursor to some elements of) the domain of neurorobotics [4, 5], which focuses on designing robotic devices that contain control systems based on or are inspired by principles of animal/human nervous systems and/or brain structures guided by the key premise that (neural) models are embodied in a body and an environment. While the gap between neurorobotics and many brain-inspired approaches largely is largely divided between focus on real-world hardware (the former) or software simulation (the latter), one pathway to bridging this gap might lie in developing powerful brain-inspired approaches that scale up to and operate robustly on problems that may ultimately be tackled by embodied robotic systems as well as using higher-quality, more realistic simulation platforms (as we do in this work). It is along this path that this work takes a step forward by developing a neurobiologically-grounded neural circuit that is used to craft a complete agent that can tackle extremely sparse reward learning control problems (tested on a more realistic, higher quality robotic system simulator), a problem that many robotic systems must ultimately face, much as humans and animals do in the real world. To build such building neural blocks and an agent system, we start from two neurocognitive theoretical foundations, predictive processing (or coding) and planning-as-inference. With respect to predictive coding, which views the brain as a type of hierarchical, pattern-creation engine [6] that engages in continual self-correction [7], we implement a fundamental circuit where each of its levels/regions are implemented by clusters of neurons that attempt to predict the state of other neural clusters/regions and adjust their synapses based on how different their predictions were from observed signals. This allows us to sidestep many of the key issues central to backprop, such as the vanishing/exploding gradient problems [8], the requirement for a long, unstable credit assignment feedback pathway [9], forward and backward locking problems [10], and the need for differentiability [11, 9]. On the other hand, motivated by planningarXiv:2209.09174v1