CAT: Contrastive Adapter Training for Personalized Image Generation