MLLM-FL: Multimodal Large Language Model Assisted Federated Learning on Heterogeneous and Long-tailed Data

Federated learning (FL), as Previous studies on federated learning (FL) often encounter performance introduced by [31], offers a solution to these privacy challenges degradation due to data heterogeneity among different by enabling collaborative model training across numerous clients clients. In light of the recent advances in multimodal large language under the orchestration of a central server, without sharing the models (MLLMs), such as GPT-4v and LLaVA, which demonstrate raw data. FL systems bring obvious advantages by involving clients their exceptional proficiency in multimodal tasks, such as downloading a global model, performing local updates using their image captioning and multimodal question answering. We introduce data, and then sending these updates back to the server. The server a novel federated learning framework, named Multimodal aggregates these updates to enhance the global model, thereby Large Language Model Assisted Federated Learning (MLLM-FL), preserving data privacy. Aside from the privacy considerations which which employs powerful MLLMs at the server end to address mentioned above, there are two fundamental acknowledgements the heterogeneous and long-tailed challenges. Owing to the about (cross-device) federated learning which have been widely advanced cross-modality representation capabilities and the extensive recognized: data heterogeneity among clients and the limited and open-vocabulary prior knowledge of MLLMs, our framework diverse computational resources on local devices [18, 30, 31, 52]. is adept at harnessing the extensive, yet previously underexploited, Data heterogeneity represents a significant challenge in federated open-source data accessible from websites and powerful server-side learning. It largely stems from the fact that the data across computational resources. Hence, the MLLM-FL not only enhances participating clients are distributed independently, with each client the performance but also avoids increasing the risk of privacy leakage having a different sample distribution.