Communication-Efficient Personalized Distributed Learning with Data and Node Heterogeneity

Abstract--T o jointly tackle the challenges of data and node heterogeneity in decentralized learning, we propose a dist ributed strong lottery ticket hypothesis (DSL TH), based on which a communication-efficient personalized learning algorithm is developed. In the proposed method, each local model is represente d as the Hadamard product of global real-valued parameters and a personalized binary mask for pruning. The local model is lea rned by updating and fusing the personalized binary masks while the real-valued parameters are fixed among different agents . T o further reduce the complexity of hardware implementatio n, we incorporate a group sparse regularization term in the los s function, enabling the learned local model to achieve struc - tured sparsity. Then, a binary mask aggregation algorithm i s designed by introducing an intermediate aggregation tenso r and adding a personalized fine-tuning step in each iteration, wh ich constrains model updates towards the local data distributi on. The proposed method effectively leverages the relativity a mong agents while meeting personalized requirements in heterog eneous node conditions. We also provide a theoretical proof for the DSL TH, establishing it as the foundation of the proposed met hod. Numerical simulations confirm the validity of the DSL TH and demonstrate the effectiveness of the proposed algorithm. Index T erms--Distributed learning, personalized learning, data and node heterogeneity, communication efficiency. As one of the most promising applications in 6G era, Artificial Intelligence of Things (AIoT) combines the artifi cial intelligence technologies with the Internet of Things (IoT) infrastructure, resembling the transformation from "conn ected things" to "connected intelligence" . This work was supported in part by National Natural Science F oundation of China under Grants 62394292 and U20A20158, Ministry of In dustry and Information Technology under Grant TC220H07E, Zhejiang Pr ovincial Key R&D Program under Grant 2023C01021, the Fundamental Resear ch Funds for the Central Universities No. 226-2024-00069, and the EU-SN S 6G CENTRIC Project. Z. Tian (email: dankotian@zju.edu.cn) was with the College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China and now is with the Center for Wireless Communications, University of Oulu, Oulu 90014, Finland. Z. Zhang (Corresponding Author, email: ning ming@zju.edu.cn) is with the College of Information Science and Electronic Engineer ing, Zhejiang University, Hangzhou 310027, China, and with the State Key L aboratory of Industrial Control Technology, Hangzhou 310027, China, and also with Zhejiang Provincial Key Laboratory of Multimodal Communic ation Networks and Intelligent Information Processing, Hangzhou 310027, China.