Federated Generative Learning with Foundation Models

Existing federated learning solutions focus on transmitting features, parameters or gadients between clients and server, which suffer from serious low-efficiency and privacy-leakage problems. Thanks to the emerging foundation generative models, we propose a novel federated learning framework, namely Federated Generative Learning, that transmits prompts associated with distributed training data between clients and server. The informative training data can be synthesized remotely based on received prompts containing little privacy and the foundation generative models. Typically, the large models are first pre-trained with massive low-quality web data for basic capability, then fine-tuned with a small number of high-quality data, especially manually-labeled data, for evoking the desired capability. Though the web data are easy to access, the high-quality training data are still scarce, since high-quality datasets are usually private or unsuitable to release. For example, most medical data are expensive to labelling and sensitive to release due to safety and privacy problems. In addition, the raw data is usually the key assets of many companies, which is impossible to be distributed. Therefore, the demand for collaborative machine learning Gong et al. (2022); Nguyen & Thai (2022); Mothukuri et al. (2021) with high efficiency and safety has become increasingly urgent. In recent years, Federated Learning (FL) McMahan et al. (2017); Kairouz et al. (2021b); Bonawitz et al. (2019); Konečnỳ et al. (2016); Silva et al. (2019); Zhao et al. (2018) has got much more attention as a promising solution to the challenges of data privacy and security in collaborative machine learning. The classic federated learning framework consists of model initialization, local training, model aggregation, and global updating.