Second, pretraining with global, unfiltered data before fine-tuning on English content can improve cultural understanding without sacrificing performance on said popular benchmarks.

Despite their demonstrated utility for NLP, multimodal counterfactual examples have been relatively unexplored due to the difficulty of creating paired image-text data with minimal counterfactual changes. To address this challenge, we introduce a scalable framework for automatic generation of counterfactual examples using text-to-image diffusion models.

Several studies have attempted to reduce the number of image-text pairs used for vision-language pre-training.

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online.