Contrastive Conditional Transport for Representation Learning

The classical contrastive loss (Oord et al., 2018; Poole et al., 2018) has achieved remarkable success in representation learning, benefiting downstream tasks in a variety of areas (Misra & Maaten, 2020; He et al., 2020; Chen et al., 2020a; Fang & Xie, 2020; Giorgi et al., 2020). The intuition of the contrastive loss is that given a query, its positive sample needs to be close, while the negative samples need to be far away in the representation space, for which the unit hypersphere is the most common assumption (Wang et al., 2017; Davidson et al., 2018). This learning scheme encourages the encoder to learn representations that are invariant to unnecessary details, and uniformly distributed on the hypersphere to maximally preserve relevant information (Hjelm et al., 2018; Tian et al., 2019; Bachman et al., 2019; Wang & Isola, 2020). A notable concern of the conventional contrastive loss is that the query's positive and negative samples are often uniformly sampled and equally treated in the comparison, which results in an inefficient estimation and limits the performance of learned representations (Saunshi et al., 2019b; Chuang et al., 2020). As illustrated in Figure 1, given a query, the conventional CL methods usually randomly take one positive sample to form the positive pair and equally treat all the other negative pairs, regardless of how informative a sample is to the query.