Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization

--Federated domain generalization (FedDG) aims to learn a globally generalizable model from decentralized clients with heterogeneous data while preserving privacy. Recent studies have introduced prompt learning to adapt vision-language models (VLMs) in FedDG by learning a single global prompt. However, such a one-prompt-fits-all learning paradigm typically leads to performance degradation on personalized samples. Although the mixture of experts (MoE) offers a promising solution for specialization, existing MoE-based methods suffer from coarse image-level expert assignment and high communication costs from parameterized routers. T o address these limitations, we propose TRIP, a T oken-level pRompt mIxture with Parameter-free routing framework for FedDG, which treats multiple prompts as distinct experts. Unlike existing image-level routing designs, TRIP assigns different tokens within an image to specific experts. T o ensure communication efficiency, TRIP incorporates a parameter-free routing mechanism based on token clustering and optimal transport. The instance-specific prompt is then synthesized by aggregating experts, weighted by the number of tokens assigned to each. Additionally, TRIP develops an unbiased learning strategy for prompt experts, leveraging the VLM's zero-shot generalization capability. Extensive experiments across four benchmarks demonstrate that TRIP achieves optimal generalization results, with communicating only 1K parameters per round. HE exponential growth of data from diverse and decentralized sources has significantly accelerated advancements in machine learning. However, traditional machine learning paradigms typically operate in a centralized manner, which requires all decentralized data to be processed on a central server, raising serious privacy concerns. This work was supported by the Shandong Provincial Natural Science Foundation under Grant ZR2020KF015, and by the Taishan Scholar Program of Shandong Province under Grant tsqn202211199 and Grant tstp20221137. S. Gong, C. Cui, and X. Dong are with the School of Computing and Artificial Intelligence, Shandong University of Finance and Economics, Jinan 250014, China (e-mail: gsh8210@163.com; X. Nie is with the School of Computer Science and Technology, Shandong Jianzhu University, Jinan 250101, China (e-mail: niexsh@hotmail.com). L. Zhu is with the College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China (e-mail: leizhu0608@gmail.com). X. Chang is with the School of Information Science and Technology, University of Science and Technology of China, Anhui 230026, China.