Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuning

-- In healthcare, federated learning (FL) is a widely adopted framework that enables privacy-preserving collaboration among medical institutions. With large foundation models (FMs) demonstrating impressive capabilities, using FMs in FL through cost-efficient adapter tuning has become a popular approach. Given the rapidly evolving healthcare environment, it is crucial for individual clients to quickly adapt to new tasks or diseases by tuning adapters while drawing upon past experiences. In this work, we introduce Federated Knowledge-Enhanced Initialization (FedKEI), a novel framework that leverages cross-client and cross-task transfer from past knowledge to generate informed initializations for learning new tasks with adapters. FedKEI begins with a global clustering process at the server to generalize knowledge across tasks, followed by the optimization of aggregation weights across clusters (inter-cluster weights) and within each cluster (intra-cluster weights) to personalize knowledge transfer for each new task. To facilitate more effective learning of the inter-and intra-cluster weights, we adopt a bi-level optimization scheme that collaboratively learns the global intra-cluster weights across clients and optimizes the local inter-cluster weights toward each client's task objective. Extensive experiments on three benchmark datasets of different modalities, including dermatology, chest X-rays, and retinal OCT, demonstrate FedKEI's advantage in adapting to new diseases compared to state-of-the-art methods. NTRODUCTION Manuscript submitted 2 Dec 2024. This Research is supported by the RIE2025 Industry Alignment Fund - Industry Collaboration Project (IAF-ICP) (Award No: I2301E0020) and Japan-Singapore Joint Call: Japan Science and T echnology Agency (JST) and Agency for Science, T echnology and Research (A*ST AR) 2024 (Award No: R24I6IR141), administered by A*ST AR (Corresponding author: Qing-song Wei (wei qingsong@ihpc.a-star.edu.sg)). EDERA TED learning (FL) has gained traction in healthcare by enabling collaborative model training across institutions without sharing sensitive data [1]. With large foundation models (FMs) demonstrating strong performance across various tasks [2], [3], integrating FMs into FL presents new opportunities for medical imaging [4]. A common approach involves fine-tuning pre-trained FMs for downstream tasks in FL [5], [6].