How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks

Recent work in continual learning has highlighted the beneficial effect of resampling weights in the last layer of a neural network ("zapping"). Although empirical results demonstrate the effectiveness of this approach, the underlying mechanisms that drive these improvements remain unclear. In this work, we investigate in detail the pattern of learning and forgetting that take place inside a convolutional neural network when trained in challenging settings such as continual learning and few-shot transfer learning, with handwritten characters and natural images. Our experiments show that models that have undergone zapping during training more quickly recover from the shock of transferring to a new domain. Furthermore, to better observe the effect of continual learning in a multi-task setting we measure how each individual task is affected. This shows that, not only zapping, but the choice of optimizer can also deeply affect the dynamics of learning and forgetting, causing complex patterns of synergy/interference between tasks to emerge when the model learns sequentially at transfer time. Despite the popularity of deep learning, neural network training is still largely considered a "dark art" (Lee et al., 2020). This alchemical connotation is in no small part due to the difficulty of building reliable intuitions about optimization in high-dimensional spaces. Analysis of neural network loss landscapes Li et al. (2018b)--the map of a network's weights to their corresponding loss values--reveals which training mechanisms are effective and helps develop new methods that account for the landscape structure. Our work builds upon the work of Javed & White (2019); Beaulieu et al. (2020); Frati et al. (2024) and sheds new light on the effect of resampling weights during pre-training, and the dynamics of learning and forgetting while navigating the complex loss landscapes of transfer (Zhuang et al., 2020) and continual (Wang et al., 2024) learning problems.