PDSL: Privacy-Preserved Decentralized Stochastic Learning with Heterogeneous Data Distribution

In the paradigm of decentralized learning, a group of agents collaborates to learn a global model using distributed datasets without a central server. However, due to the heterogeneity of the local data across the different agents, learning a robust global model is rather challenging. Moreover, the collaboration of the agents relies on their gradient information exchange, which poses a risk of privacy leakage. In this paper, to address these issues, we propose PDSL, a novel privacy-preserved decentralized stochastic learning algorithm with heterogeneous data distribution. On one hand, we innovate in utilizing the notion of Shapley values such that each agent can precisely measure the contributions of its heterogeneous neighbors to the global learning goal; on the other hand, we leverage the notion of differential privacy to prevent each agent from suffering privacy leakage when it contributes gradient information to its neighbors. We conduct both solid theoretical analysis and extensive experiments to demonstrate the efficacy of our PDSL algorithm in terms of privacy preservation and convergence. I NTRODUCTION Distributed machine learning refers to a family of algorithms aiming at learning from data distributed among multiple agents [1]-[3]. Based on communication topology, approaches to designing distributed learning algorithms include centralized learning and decentralized learning . As a popular approach for centralized learning, Federated Learning (FL) enables the central server to organize the collaborative training process through the interaction of model parameters while training data can be stored locally at the agents [4], [5]. However, this setup requires constant communication with the central server, potentially creating a bottleneck. To mitigate this concern, some decentralized learning algorithms have been proposed such that agents communicate their updates with their neighbors in a communication network without the assistance of a central server.