TACO: Tackling Over-correction in Federated Learning with Tailored Adaptive Correction

T ACO: Tackling Over-correction in Federated Learning with Tailored Adaptive Correction Weijie Liu 1,2, Ziwei Zhan 1, Carlee Joe-Wong 3, Edith Ngai 2, Jingpu Duan 4, Deke Guo 1, Xu Chen 1, Xiaoxi Zhang 1 1 Sun Y at-sen University, 2 The University of Hong Kong, 3 Carnegie Mellon University, 4 Pengcheng Laboratory Email: liuwj0817@connect.hku.hk, Abstract --Non-independent and identically distributed (Non-IID) data across edge clients have long posed significant challenges to federated learning (FL) training. Prior works have proposed various methods to mitigate this statistical heterogeneity. While these methods can achieve good theoretical performance, they may lead to the over-correction problem, which degrades model performance and even causes failures in model convergence. In this paper, we provide the first investigation into the hidden over-correction phenomenon brought by the uniform model correction coefficients across clients adopted by the existing methods. T o address this problem, we propose T ACO, a novel algorithm that addresses the non-IID nature of clients' data by implementing fine-grained, client-specific gradient correction and model aggregation, steering local models towards a more accurate global optimum. Moreover, we verify that leading FL algorithms generally have better model accuracy in terms of communication rounds rather than wall-clock time, resulting from their extra computation overhead imposed on clients. T o enhance the training efficiency, T ACO deploys a lightweight model correction and tailored aggregation approach that requires minimum computation overhead and no extra information beyond the synchronized model parameters. T o validate T ACO's effectiveness, we present the first FL convergence analysis that reveals the root cause of over-correction.