Real-world data for classification is often labeled by multiple annotators. For analyzing such data, we introduce CROWDLAB, a straightforward approach to utilize any trained classifier to estimate: (1) A consensus label for each example that aggregates the available annotations; (2) A confidence score for how likely each consensus label is correct; (3) A rating for each annotator quantifying the overall correctness of their labels. Existing algorithms to estimate related quantities in crowdsourcing often rely on sophisticated generative models with iterative inference. CROWDLAB instead uses a straightforward weighted ensemble. Existing algorithms often rely solely on annotator statistics, ignoring the features of the examples from which the annotations derive. CROWDLAB utilizes any classifier model trained on these features, and can thus better generalize between examples with similar features. On real-world multi-annotator image data, our proposed method provides superior estimates for (1)-(3) than existing algorithms like Dawid-Skene/GLAD.

A often provide imperfect labels. It is thus common very general approach, ActiveLab can be used: with any to employ multiple annotators to label data type of classifier model (or ensemble of multiple models) with some overlap between their examples. We and data modality, for active learning with multiple annotators study active learning in such settings, aiming to where the set of annotators changes over time, for train an accurate classifier by collecting a dataset traditional active learning where each example is labeled with the fewest total annotations. Here we propose at most once (Appendix D), and for active label cleaning ActiveLab, a practical method to decide what where all data is already labeled by at least one annotator to label next that works with any classifier model and the goal is to establish the highest quality consensus and can be used in pool-based batch active learning labels within a limited annotation budget. ActiveLab is with one or multiple annotators.