Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift