Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval

There is a scarcity of multilingual visionlanguage models that properly account for the perceptual differences that are reflected in image captions across languages and cultures. In this work, through a multimodal, multilingual retrieval case study, we quantify the existing lack of model flexibility. We empirically show Figure 1: Example perception differences between native performance gaps between training on captions English and German speakers. Examples are captions that come from native German perception from Flickr30K (Young et al., 2014) and Multi30K and captions that have been either machinetranslated (Elliott et al., 2016). Note differences in mentioned objects or human-translated from English ("sand arena", "parasol") and specificity ("Heurigen into German. To address these gaps, we further bench" vs. "table", "horse" vs. "bronco"). German propose and evaluate caption augmentation captions here are translated to English.