Rehearsal-Free Continual Federated Learning with Synergistic Regularization

Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods. Federated learning (FL) is to facilitate the collaborative training of a global deep learning model among multiple edge clients while ensuring the privacy of their locally stored data (McMahan et al., 2017; Wang et al., 2023a; Liu et al., 2024). Recently, FL has garnered significant interest and found applications in diverse domains, including recommendation systems (Yang et al., 2020; Li et al., 2024d) and smart healthcare solutions (Xu et al., 2021; Nguyen et al., 2022). Typically, FL has been studied in a static setting, where the number of training samples does not change over time.