Contrary to prior assumptions, we identify extraction of weights, not extraction of weight signs, as the critical bottleneck.

Provably Robust Deep Learningvia Adversarially Trained Smoothed Classifiers.

S2.2 Varianceofimportanceweights The importance-sampled estimate of the log-likelihood used to retrain the oracle (Equation 17) is unbiased, butmayhavehighvariance duetothevariance oftheimportance weights. LetLβ: X R Rdenote a pertinent loss function induced by the oracle parameters,β, (e.g., the squared errorLβ(x,y) = (Eβ[y |x] y)2). While the bound,L, on Lβ may be restrictive in general, for any givenapplication one may beable touse domain-specific knowledge toestimateL. CbAS naturally controls the importance weight variance. Design procedures that leverage a trust region can naturally bound thevariance oftheimportance weights.

In the traditional supervised learning framework, active learning enables the learner to actively engage in the construction of the labeled training set by selecting a fixed-sized subset of unlabeled examples for labeling by an oracle, where the number of labels requested is referred to as the budget .

Y The constraint P X,P Y Prob(X) states that P X and P Y are valid probability distributions. For brevity, we shall ignore explicitly stating it in the rest of the proof. The above equation is similar in spirit to the Kantrovich-Rubinstein duality. An important observation to note is that the above optimization only maximizes over a single discriminator function (as opposed to two functions in optimization (2)). Hence, it is easier to train it in large-scale deep learningproblemssuchasGANs.