However, existing theories fall short in providing explanations for this phenomenon.

Eqn. 3) is equivalent to optimizing CL (InfoNCE, cf. To complete the proof, we start with giving some important notations and theorem. Here we simply disregard the constant term present in Eqn. 4 as it does not impact optimization, and From the Thm.3.2, we have the equivalence between InfoNCE and CL-DRO. By using McDiarmid's inequality in Thm A.4,for any ϵ, we have: While Corollary 3.4 has already been proven in [ We start with introducing a useful lemma. Then the CL-DRO objective is the tight variational estimation of ϕ -divergence.

This study reveals the inherent tolerance of contrastive learning (CL) towards sampling bias, wherein negative samples may encompass similar semantics (\eg labels). However, existing theories fall short in providing explanations for this phenomenon. We bridge this research gap by analyzing CL through the lens of distributionally robust optimization (DRO), yielding several key insights: (1) CL essentially conducts DRO over the negative sampling distribution, thus enabling robust performance across a variety of potential distributions and demonstrating robustness to sampling bias; (2) The design of the temperature $\tau$ is not merely heuristic but acts as a Lagrange Coefficient, regulating the size of the potential distribution set; (3) A theoretical connection is established between DRO and mutual information, thus presenting fresh evidence for ``InfoNCE as an estimate of MI'' and a new estimation approach for $\phi$-divergence-based generalized mutual information. We also identify CL's potential shortcomings, including over-conservatism and sensitivity to outliers, and introduce a novel Adjusted InfoNCE loss (ADNCE) to mitigate these issues. It refines potential distribution, improving performance and accelerating convergence. Extensive experiments on various domains (image, sentence, and graphs) validate the effectiveness of the proposal. The code is available at \url{https://github.com/junkangwu/ADNCE}.