Neuroplastic Expansion in Deep Reinforcement Learning

In the realm of neuroscience, it has been observed that biological agents often experience a diminishing ability to adapt over time, analogous to the gradual solidification of neural pathways in the brain (Livingston, 1966). This phenomenon, typically known as the loss of plasticity (Mateos-Aparicio and Rodríguez-Moreno, 2019), significantly affects an agent's capacity to learn continually, especially when agents learn by trial and error in deep reinforcement learning (deep RL) due to the nonstationarity nature. The declining adaptability throughout the learning process can severely hinder the agent's ability to effectively learn and respond to complex or non-stationary scenarios (Abbas et al., 2023). This limitation presents a fundamental obstacle to achieving sustained learning and adaptability in artificial agents, which echoes the plasticity-stability dilemma (Abraham and Robins, 2005) observed in biological neural networks. There have been several recent studies highlighting a significant loss of plasticity in deep RL (Kumar et al., 2020, Lyle et al., 2022), which substantially restricts the agent's ability to learn from subsequent experiences (Lyle et al., 2023, Ma et al., 2023). The identification of primacy bias (Nikishin et al., 2022) further illustrates how agents may become overfitted to early experiences, which inhibits learning from subsequent new data. The consequences of plasticity loss further impede deep RL in continual learning scenarios, where the agent struggles to sequentially learn across a series of different tasks (Dohare et al., 2024). 1