A Task Details

ViL T for each task, and details about how low-shot versions of each task are sampled. B.1 Applying ViL T to Multi-Choice T asks B.1.1 Applying ViL T to VCR We follow previous work [Zellers et al., 2021, Hessel et al., 2022] and draw colored boxes directly The grounded text references, e.g. We follow the original implementations [Zellers et al., 2019b, Bisk et al., 2020] to model these tasks, B.2 Applying ViL T to Unimodal T asks We conduct low-shot experiments to test the model's transferability to unimodal However, different sub-samples the training set may lead to different results. For vision-only tasks, we found that simply using "This is an image." We also conduct ablation studies that include two baselines: (1) not inputting any image to ViL T at all, and (2) inputting the zero-vector image instead of the average image of the COCO dataset.