Ensembling has a long history in statistical data analysis, with many impactful applications. However, in many modern machine learning settings, the benefits of ensembling are less ubiquitous and less obvious.

Figure 2: movedby thedirection k (left) andbottom-k (right) histogram givenstandard mode. Figure for k (left) k histogram giv mode.

AI models are increasingly prevalent in high-stakes environments, necessitating1 thorough assessment oftheir capabilities and risks.

Convergence of stochastic gradient descent for non-smooth problems is a known result. For completeness, wereproduce and adapt ausual proof toour setting. Let us denote byF the class of functions fromX toY we are going to work with. Assumption 1 states that we have a well-specified modelF to estimate the median,i.e. Let us begin by controlling the estimation error.

The workhorse of machine learning is stochastic gradient descent. To access stochastic gradients, it is common to consider iteratively input/output pairs of a training dataset. Interestingly, it appears that one does not need full supervision to access stochastic gradients, which is the main motivation of this paper. After formalizing the"activelabeling" problem, whichfocuses onactivelearningwith partial supervision, we provide a streaming technique that provably minimizes the ratio of generalization error over the number of samples.