Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boosting

One-shot Federated Learning (OFL) has become a promising learning paradigm, enabling the training of a global server model via a single communication round. In OFL, the server model is aggregated by distilling knowledge from all client models (the ensemble), which are also responsible for synthesizing samples for distillation. In this regard, advanced works show that the performance of the server model is intrinsically related to the quality of the synthesized data and the ensemble model. To promote OFL, we introduce a novel framework, Co-Boosting, in which synthesized data and the ensemble model mutually enhance each other progressively. Specifically, Co-Boosting leverages the current ensemble model to synthesize higher-quality samples in an adversarial manner. These hard samples are then employed to promote the quality of the ensemble model by adjusting the ensembling weights for each client model. Consequently, Co-Boosting periodically achieves high-quality data and ensemble models. Extensive experiments demonstrate that Co-Boosting can substantially outperform existing baselines under various settings. Moreover, Co-Boosting eliminates the need for adjustments to the client's local training, requires no additional data or model transmission, and allows client models to have heterogeneous architectures. Federated learning (FL) (McMahan et al., 2017) has emerged as a prominent distributed machine learning framework to train a global server model via collaboration among users without sharing their dataset. Though the multi-round parameter-server communication paradigm offers the benefit of effectively exchanging information among clients and the central server, it might not be feasible in the real world. This paradigm brings forth significant challenges: 1) heavy communication burden and the risk of connection drop errors between clients and the server (Li et al., 2020a; Kairouz et al., 2021; Dai et al., 2022), and 2) potential risk for man-in-the-middle attacks (Wang et al., 2021) and various other privacy or security concerns (Mothukuri et al., 2021; Yin et al., 2021). One-shot FL (OFL) (Guha et al., 2019) has emerged as a solution to these issues by restricting communication rounds to a single iteration, thereby mitigating errors arising from multi-round communication and concurrently diminishing the vulnerability to malicious interception. Furthermore, OFL is more practical, particularly within contemporary model market scenarios (Vartak et al., 2016) where clients predominantly offer pre-trained models.