Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP [60]. In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions. Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors. Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learning to learn low-dimensional and informative representations, bridging theoretical insights and practical performance.

The rapid progress in generative models has resulted in impressive leaps in generation quality, blurring the lines between synthetic and real data. Web-scale datasets are now prone to the inevitable contamination by synthetic data, directly impacting the training of future generated models. Already, some theoretical results on self-consuming generative models (a.k.a., iterative retraining) have emerged in the literature, showcasing that either model collapse or stability could be possible depending on the fraction of generated data used at each retraining step. However, in practice, synthetic data is often subject to human feedback and curated by users before being used and uploaded online. For instance, many interfaces of popular text-to-image generative models, such as Stable Diffusion or Midjourney, produce several variations of an image for a given query which can eventually be curated by the users. In this paper, we theoretically study the impact of data curation on iterated retraining of generative models and show that it can be seen as an implicit preference optimization mechanism.

We consider the problem of estimating a rank-1 signal from a noisy data matrix.

A classical problem that fits in the framework of (1) is the best subset selection problem in linear regression [6,20].

Last, weprovide numerical experiments toillustrate our results and endorse our intuitions.

We attain the optimal eO( T) regret when the dynamics are unknown to the learner, and poly(logT) regret when known, provided that the cost functions are strongly convex (as in LQR).

We consider logistic regression, and more generally, linear classification, in the streaming model. In our setting, we are given a dataset consisting ofT examples (xt,yt), where t [T], xt Rd, yt { 1,1}. The examples arrive one by one, and moreover, the nonzero coordinates of each examplext arrive one by one.

Ouranalysis culminates inshowing thatthere exists a universally consistent histogram-style estimator that converges to any multi-view model with a finite number of Lipschitz continuous components at a rate of eO(1/3 n) in L1 error.

There exists constantsC1,C2,C3,C4 depending onψ such that the following holds true.