MIT researchers find 'systematic' shortcomings in ImageNet data set

MIT researchers have concluded that the well-known ImageNet data set has "systematic annotation issues" and is misaligned with ground truth or direct observation when used as a benchmark data set. "Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for," the researchers write in a paper titled "From ImageNet to Image Classification: Contextualizing Progress on Benchmarks." "We believe that developing annotation pipelines that better capture the ground truth while remaining scalable is an important avenue for future research." When the Stanford University Vision Lab introduced ImageNet at the Conference on Computer Vision and Pattern Recognition (CVPR) in 2009, it was much larger than many previously existing image data sets. The ImageNet data set contains millions of photos and was assembled over the span of more than two years. ImageNet uses the WordNet hierarchy for data labels and is widely used as a benchmark for object recognition models.