We derive non-asymptotic spectral bands that bound the squared InfoNCE gradient norm via alignment, temperature, and batch spectrum, recovering the 1/τ2 law and closely tracking batch-mean gradients on synthetic data and ImageNet.

However, there still exist many properties of contrastive learning that are not guaranteed.

We also conduct empirical experiments to verify the effectiveness of those perturbations. As shown in Fig. A1, all of the perturbed text-features In addition, now that every perturbation can directly produce the description ( i.e., text-feature) of And the results are shown in Tab. OOD performance when the ID data is shifted. Table A2: Additionally improved ID accuracy on shifted datasets. Fig. A2, compared to the shifted ImageNet-A [ Sketch only preserve objects' shape and main texture, while the color information is totally vanished.

In this subsection, we derive the necessary and sufficient condition in proposition??. Denote B,K be some natural numbers. We introduce the proposition from [8] that proves geometrical convergence of positive concave mapping. Bycorollary 2, g(v(n);Q) is a concave mapping. Wedonotapplyweightdecayanduse cosine scheduled the learning rate.

Existing work focuses on ahandful ofcurated generativeprocesses which require expert knowledge to design, making it hard to scale up. To overcome this, we propose training with alarge dataset of twenty-one thousand programs, each one generating adiverse setofsynthetic images.