A Simple Illustration of Interleaved Learning using Kalman Filter for Linear Least Squares

IL is one of the mechanisms expounded by Complementary Learning Systems Theory (McClelland, McNaughton and O'Reilly, 1995; Marr, 1971) on how successful learners such as human beings mitigate effects of'catastrophic interference' while learning. Recent illustrations of IL using neural networks include Saxena, Shobe and McNaughton, 2022, who exhibited that if the new information is similar to a subset of old items, then deep neural networks can learn the new information rapidly and with the same level of accuracy by interleaving the old items in the subset. A similar insight was presented in McClelland, McNaughton and Lampinen, 2020, where it was shown that for artificial neural networks, information consistent with prior knowledge can sometimes be integrated very quickly. Another recent paper (Ban and Xie, 2021) formulated interleaved machine learning as a multi-level optimization problem, and developed an efficient differentiable algorithm to solve the interleaving learning problem with application to neural architecture search. A closely related biological concept is interleaved replay which also has been empirically validated in the literature (Gepperth and Karaoguz, 2016; Kemker and Kanan, 2018). Over the past couple of decades, ideas inspired by biological IL have been utilized in a wide array of online learning methods as well, especially to prevent catastrophic forgetting. See, for example Wang et.