A Convergence Theory for Federated Average: Beyond Smoothness

With the growing of computational power on edge devices, such as mobile phones, wearable devices, smart watches, self-driving cars, and so on, developing distributed optimization methods to address the needs of those applications is increasingly demanded. There are three core challenges existing in the distributed computing applications, including expensive communication, privacy concerns, and heterogeneity. To tackle the above-mentioned challenges, federated learning (FL) has emerged as an important paradigm in today's machine learning for distributed learning that enables different clients (also known as nodes) to collaboratively learn a model while keeping their private data. To train an FL algorithm in a distributed manner, the clients must transmit their training parameters to a central server. Typically, the central server has the same model architecture as the local clients. Similar to centralized parallel optimization, FL lets the clients do most of the computation while the central server updates the model parameters using the descending directions returned by the local clients. However, learning with FL significantly differs from the traditional parallel optimization in distributed learning in the various needs, including piracy requirements, large-scale machine learning and efficiency.