Beyond Disagreement-Based Agnostic Active Learning

We study agnostic active learning, where the goal is to learn a classifier in a prespecified hypothesis class interactively with as few label queries as possible, while making no assumptions on the true function generating the labels. The main algorithm for this problem is disagreement-based active learning, which has a high label requirement. Thus a major challenge is to find an algorithm which achieves better label complexity, is consistent in an agnostic setting, and applies to general classification problems. In this paper, we provide such an algorithm. Our solution is based on two novel contributions; first, a reduction from consistent active learning to confidence-rated prediction with guaranteed error, and second, a novel confidence-rated predictor.