Finetuning CLIP to Reason about Pairwise Differences

Vision-language models (VLMs) such as CLIP are trained via contrastive learning between text and image pairs, resulting in aligned image and text embeddings that are useful for many downstream tasks. A notable drawback of CLIP, however, is that the resulting embedding space seems to lack some of the structure of their purely text-based alternatives. For instance, while text embeddings have been long noted to satisfy analogies in embedding space using vector arithmetic, CLIP has no such property. In this paper, we propose an approach to natively train CLIP in a contrastive manner to reason about differences in embedding space. We finetune CLIP so that the differences in image embedding space correspond to text descriptions of the image differences, which we synthetically generate with large language models on image-caption paired datasets. We first demonstrate that our approach yields significantly improved capabilities in ranking images by a certain attribute (e.g., elephants are larger than cats), which is useful in retrieval or constructing attribute-based classifiers, and improved zeroshot classification performance on many downstream image classification tasks. In addition, our approach enables a new mechanism for inference that we refer to as comparative prompting, where we leverage prior knowledge of text descriptions of differences between classes of interest, achieving even larger performance gains in classification. Finally, we illustrate that the resulting embeddings obey a larger degree of geometric properties in embedding space, such as in text-to-image generation. Vision-language models (VLMs) (Jia et al., 2021; Li et al., 2022a), and more specifically CLIP (Radford et al., 2021), leverage paired instances of images and corresponding text descriptions to produce a general-purpose joint embedding between images and language. These models have created a new paradigm of prompting (Radford et al., 2021; Li & Liang, 2021; Bach et al., 2022). In this new paradigm, we can easily design image classifiers through text descriptions of classes and by selecting which of our class descriptions most closely aligns with an image (in terms of cosine similarity in the multimodal embedding space). These models can also generate images corresponding to user-specified text prompts (Podell et al., 2023). Ultimately, this paradigm fundamentally relies on the accurate alignment of image and text modalities. While contrastive-based pretraining on large datasets aims to achieve this embedding alignment, a notable drawback of CLIP models is that they do not exhibit the structure of purely languagebased embeddings. For instance, text embeddings satisfy analogies in embedding space using vector arithmetic, e.g., Text("King") - Text("Man") + Text("Woman") Text ("Queen") (Mikolov et al., 2013), while CLIP has no such property.