Wealsoshow thatacertified version ofthecomputationally tractable DOO algorithm matches these packing and integral bounds.

Notation and conventions: [n] = {1,,n}; |S| is the size of setP; dxe is the smallest integer greater thanx; bxc is the largest integer smaller thanx; kXk is the nuclear norm of matrixL, i.e., the sum of the singular values of matrixX; Z+ is the set of positive integers;Z i is the set of integers which are greater thani; Given S1, S2, the reduction ofS1 by S2 is denoted as S1\S2={i S1:i / S2};finally,A(n) B(n)meansA(n)/B(n) 1asn .

Wealsousestandard "Big O" notations ( ), !( ), e.g., see [CLRS09, Chapter 3]. In (a),"(4) 0.2" means 4 classesandkµk2/ p p=0.2.

Note thatppos is task-specific; here we use the class oracle,i.e. the ImageNet-100 labels,todefinethepositivesamples. In Figure 1, we plot theproxy task performance, i.e. the percentage of queries where the key is ranked over all negatives, across training for MoCo [19], MoCo-v2 [10] and some variants inbetween. As mentioned above, all results in Figure1areforthesameτ =0.2. Ablations showed that this yields at best performance as good as mixingwiththequery,butonaverageabout0.1-0.2%lower. This weighing scheme also resulted in slightly inferior results.

Wetherefore startbydelving deeper intoatop-performing frameworkandshow evidence that harder negatives are needed to facilitate better and faster learning.

Our rule may be understood as ageneralization of contrastive Hebbian learning to meta-learning and notably, it neither requires computing second derivativesnorgoing backwardsintime,twocharacteristic features of previous gradient-based methods that are hard to conceive in physicalneuralcircuits.