FedDec: Peer-to-peer Aided Federated Learning

Federated learning (FL) is a recent machine learning framework that allows multiple agents, each of them with their own dataset, to train a model collaboratively without sharing their data [1-4]. The federated setting assumes that all agents are connected to a server that can communicate with each of them and that is in charge of aggregating the agents' updates to obtain the global model. This is similar to parallel distributed (PD) model training [5-8], with one crucial difference: in the latter, the agents send gradients to the central server to update the parameter value with a gradient step, while in FL the agents send their own local parameters for the server to average them. This has an impact on the communication frequency required by each framework: in PD one round of communication between (usually all) the agents and the server has to happen every time a (mini-batch) stochastic gradient descent (SGD) step is taken at the nodes, while in FL (i) multiple SGD updates can happen before a new server communication round takes place (which in FL literature are usually called local updates), and (ii) not all devices need to engage in the server communication round (which is known as partial participation). This makes FL a much more suitable option for settings with a large number of agents and a limited communication bandwidth with the server. In contrast to the approaches described above, the decentralized setting does not rely on a central server for the aggregation of the nodes' updates.