Lorica: A Synergistic Fine-Tuning Framework for Advancing Personalized Adversarial Robustness

Abstract--The growing use of large pre-trained models in edge computing has made model inference on mobile clients both feasible and popular . Y et these devices remain vulnerable to adversarial attacks, threatening model robustness and security. Federated adversarial training (F A T) offers a promising solution by enhancing robustness while preserving client privacy. However, F A T often yields a generalized global model that struggles with heterogeneous client data, leading to limited personalization and significant communication overhead. In this paper, we propose Lorica, a personalized synergistic adversarial training framework that delivers customized defense models through a two-phase process. In Phase 1, Lorica applies LoRA-F A for local adversarial fine-tuning, enabling personalized robustness while reducing communication by uploading only LoRA-F A parameters. In Phase 2, a forward-gating selection strategy improves benign accuracy, further refining the personalized model. This yields tailored defense models that effectively balance robustness and accuracy. Extensive experiments on benchmark datasets demonstrate that Lorica can achieve up to 68 improvements in communication efficiency compared to state-of-the-art algorithms, while achieving up to 29.9% and 52.2% enhancements in adversarial robustness and benign accuracy, respectively. Index T erms--Pre-trained models, personalized federated learning, adversarial training, fine-tuning. With the rapid advancement of large language models (LLM), large-scale pre-trained models have garnered widespread attention across various fields, including computer vision [1] and autonomous driving [2], etc. Fine-tuning pre-trained models for downstream tasks has gradually established itself as a novel learning paradigm [3]. Meanwhile, the increasing computational power of edge devices has facilitated the localized deployment of the pre-trained models, unlocking their potential for various applications on devices [4]. However, recent studies have revealed substantial security risks associated with deploying pre-trained models on edge devices. T. Qi, and L. Xue are with the School of Cyber Science and Technology, Sun Y at-sen University, Shenzhen, China. Zhan is with the School of Automation, Beijing Institute of Technology, Beijing, China. X. Ma is with the School of Cyber Science and Engineering, Xi'an Jiaotong University, Xi'an, China. We also thank the Guangdong Key Laboratory of Information Security Technology for their support.