Towards efficient compression and communication for prototype-based decentralized learning

In prototype-based federated learning, the exchange of model parameters between clients and the master server is replaced by transmission of prototypes or quantized versions of the data samples to the aggregation server. A fully decentralized deployment of prototype-based learning, without a central agregartor of prototypes, is more robust upon network failures and reacts faster to changes in the statistical distribution of the data, suggesting potential advantages and quick adaptation in dynamic learning tasks, e.g., when the data sources are IoT devices or when data is non-iid. In this paper, we consider the problem of designing a communication-efficient decentralized learning system based on prototypes. We address the challenge of prototype redundancy by leveraging on a twofold data compression technique, i.e., sending only update messages if the prototypes are informationtheoretically useful (via the Jensen-Shannon distance), and using clustering on the prototypes to compress the update messages used in the gossip protocol. We also use parallel instead of sequential gossiping, and present an analysis of its age-of-information (AoI). Our experimental results show that, with these improvements, the communications load can be substantially reduced without decreasing the convergence rate of the learning algorithm. Federated Learning (FL) [1], [2], [3] and Decentralized Federated Learning (DFL) [4], [5] provide good approaches for distributed machine learning system where the main focus is the minimization of a global loss function using different versions of a model created by multiple clients. These approaches have been extensively studied in the literature and applied, traditionally, to process private data in areas such as health and banking. In this paper, differently to these well-known approaches, we focus on the analysis and implementation of a decentralized machine learning system based on prototypes. On the one hand, our choice of prototype-based algorithms is motivated by the advantages of these prototypes as compact representation of the data, capturing the essential features and patterns within the dataset.