r/MachineLearning - DeepMind's new neural network model beats AlexNet with 13 images per class

Definitely does have "echos" of BERT and friends from the NLP side of things, though still has a while to go to reach a similarly large revolution in performance. However, the OP's title does not match the claims of the paper. With unsupervised pretraining on over 1M images 13 labels per class they get 64% top-5 accuracy, well below Alexnet's 82% accuracy. While the paper's investigation is pretty thorough, I don't think they mention either compute requirements (given it's deepmind, I would default to assuming it's gigantic) or how the approach scales with different amounts of unsupervised data. Like how does it perform if only training the CPC feature extractor on half of imagenet?