Flexible Personalized Split Federated Learning for On-Device Fine-Tuning of Foundation Models

--Fine-tuning foundation models is critical for superior performance on personalized downstream tasks, compared to using pre-trained models. Collaborative learning can leverage local clients' datasets for fine-tuning, but limited client data and heterogeneous data distributions hinder effective collaboration. T o address the challenge, we propose a flexible personalized federated learning paradigm that enables clients to engage in collaborative learning while maintaining personalized objectives. Given the limited and heterogeneous computational resources available on clients, we introduce flexible personalized split federated learning (FlexP-SFL). Based on split learning, FlexP-SFL allows each client to train a portion of the model locally while offloading the rest to a server, according to resource constraints. Additionally, we propose an alignment strategy to improve personalized model performance on global data. Experimental results show that FlexP-SFL outperforms baseline models in personalized fine-tuning efficiency and final accuracy. Foundation models, such as GPT [1], [2] and BERT [3], as well as more recent architectures [4]-[7], are large-scale machine learning models pre-trained on vast and diverse datasets [8]. These models are designed to capture broad and generalizable patterns across multiple domains, enabling strong performance on a wide range of tasks with minimal adaptation.