ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

A often provide imperfect labels. It is thus common very general approach, ActiveLab can be used: with any to employ multiple annotators to label data type of classifier model (or ensemble of multiple models) with some overlap between their examples. We and data modality, for active learning with multiple annotators study active learning in such settings, aiming to where the set of annotators changes over time, for train an accurate classifier by collecting a dataset traditional active learning where each example is labeled with the fewest total annotations. Here we propose at most once (Appendix D), and for active label cleaning ActiveLab, a practical method to decide what where all data is already labeled by at least one annotator to label next that works with any classifier model and the goal is to establish the highest quality consensus and can be used in pool-based batch active learning labels within a limited annotation budget. ActiveLab is with one or multiple annotators.