Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation

Federated learning aims to train a global model in a distributed environment that is close to the performance of centralized training. Most existing methods overlook the scenario where only a small portion of clients participate in training within a large-scale client setting, whereas our experiments show that this scenario presents a more challenging federated learning task. Therefore, we propose a Knowledge Distillation with teacher-student Inequitable Aggregation (KDIA) strategy tailored to address the federated learning setting mentioned above, which can effectively leverage knowledge from all clients. In KDIA, the student model is the average aggregation of the participating clients, while the teacher model is formed by a weighted aggregation of all clients based on three frequencies: participation intervals, participation counts, and data volume proportions. During local training, self-knowledge distillation is performed. Additionally, we utilize a generator trained on the server to generate approximately independent and identically distributed (IID) data features locally for auxiliary training. We conduct extensive experiments on the CIFAR-10/100/CINIC-10 datasets and various heterogeneous settings to evaluate KDIA. The results show that KDIA can achieve better accuracy with fewer rounds of training, and the improvement is more significant under severe heterogeneity. Corresponding author Email address: maxxldnc@163.com Introduction In recent years, deep learning has been widely applied in industrial production and people's daily lives, and the nourishment of deep learning--big data has been fully utilized. However, with the growing emphasis on data privacy, the problem of data silos has severely hindered the further development of deep learning [1]. To address this, McMahan et al. proposed the Federated Learning framework and the FedAvg algorithm [2], where clients can collaboratively train a global model with the server without sharing their local data.