Robust Fine-tuning of Zero-shot Models via Variance Reduction