Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation

Machine learning models can perform poorly when the train and test data are drawn from different distributions, which is especially troublesome for performance-critical applications such as image recognition for self-driving cars (Yu et al., 2020; Sun et al., 2020) or medical image diagnosis (AlBadawy et al., 2018; Dai & Gool, 2018). In this work, we study the unsupervised domain adaptation (UDA) setting where we have access to labeled data from a source domain and unlabeled data from a target domain, and the goal is to get high accuracy on the target domain. Conventional algorithms for UDA aim to learn domain-invariant features (Tzeng et al., 2014; Ganin et al., 2016; Tzeng et al., 2017; Shu et al., 2018; Sun et al., 2019)--intuitively, if the distributions over features for the source and target domains are indistinguishable and the accuracy is high on the source, then the accuracy should be high on the target as well. This is typically intuitively justified by theoretical notions such as H H-divergence, which measures the distinguishability of source and target feature spaces (Ben-David et al., 2010). However, Zhao et al. (2019) show that domain invariance is not sufficient for target generalization, and thus some recent works have begun to develop principled algorithms for domain adaptation (Kumar et al., 2020; Wei et al., 2021; Cai et al., 2021). In this paper, we find that a surprisingly simple and effective method for UDA is out-of-the-box contrastive pretraining on source and target unlabeled data, followed by fine-tuning on source labeled data. In our experiments, contrastive pre-training obtains comparable or better results to strong UDA methods based on domain adversarial neural networks (Ganin et al., 2016; Shu et al., 2018) and self-training (Prabhu et al., 2021) on visual adaptation benchmarks including DomainNet, BREEDS Living-17, BREEDS Entity-30, and STL-10 CIFAR-10 (results in Table 2). However, we show that contrastive pre-training diverges from conventional UDA intuitions and learns features that are easily separable between domains; for example in the learned feature space in DomainNet, we can predict the domain of an image with only 8% error, which is much lower than in the DANN feature space (14%)--see Table 1 in Section 4. In fact, in the contrastive pre-trained feature space for DomainNet, it is as easy to distinguish betweeen two domains as it is to distinguish between two classes.