The Visual Task Adaptation Benchmark

Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets. Y et, the absence of a unified yardstick to evaluate general visual representations hinders progress. Many sub-fields promise representations, but each has different evaluation protocols that are either too constrained (linear classification), limited in scope (ImageNet, CIFAR, Pascal-VOC), or only loosely related to representation quality (generation). We present the Visual Task Adaptation Benchmark (VT AB): a diverse, realistic, and challenging benchmark to evaluate representations. VT AB embodies one principle: good representations adapt to unseen tasks with few examples . We run a large VT AB study of popular algorithms, answering questions like: How effective are ImageNet representation on nonstandard datasets? Is self-supervision useful if one already has labels? Deep learning has revolutionized computer vision. Distributed representations learned from ...