ClipGrader: Leveraging Vision-Language Models for Robust Label Quality Assessment in Object Detection

A BSTRACT High-quality annotations are essential for object detection models, but ensuring label accuracy -- especially for bounding boxes -- remains both challenging and costly. This paper introduces ClipGrader, a novel approach that leverages vision-language models to automatically assess the accuracy of bounding box annotations. By adapting CLIP (Contrastive Language-Image Pre-training) to evaluate both class label correctness and spatial precision of bounding box, ClipGrader offers an effective solution for grading object detection labels. Tested on modified object detection datasets with artificially disturbed bounding boxes, Clip-Grader achieves 91% accuracy on COCO with a 1.8% false positive rate. Moreover, it maintains 87% accuracy with a 2.1% false positive rate when trained on just 10% of the COCO data. Our experiments demonstrate ClipGrader's ability to identify errors in existing COCO annotations, highlighting its potential for dataset refinement. When integrated into a semi-supervised object detection (SSOD) model, ClipGrader readily improves the pseudo label quality, helping achieve higher mAP (mean Average Precision) throughout the training process. ClipGrader thus provides a scalable AIassisted tool for enhancing annotation quality control and verifying annotations in large-scale object detection datasets. In object detection, a fundamental task in computer vision, the accuracy of annotations which encompasses both the correctness of the class label and spatial precision of the bounding box is crucial. However, curating high-quality object detection datasets is a significant challenge due to the time-consuming and expensive nature of manual annotation processes, not to mention the inevitability of errors creeping in (Kuznetsova et al., 2018; V ondrick et al., 2013). Existing approaches to data collection and annotation, such as crowd sourcing, web scraping, or AI-generated labels, often introduce noise and inconsistencies, potentially compromising model performance (Papadopoulos et al., 2016; Northcutt et al., 2021; Zare & Y azdi, 2022). With the increasing complexity and scale of datasets, traditional methods of quality control such as manual reviews or simple heuristics struggle to meet the demand.