Fine-Tuning Personalization in Federated Learning to Mitigate Adversarial Clients