Big Is Better: Is Deep Learning Rigged Against Smaller Labs?

It has been recently established with OpenAI's GPT-3 release that larger models perform better. Not just GPT, but other NLP models like T5 too have given better results compared to previous works. Historically, NLP systems have struggled to learn from a few examples. But, with GPT-3, the researchers showed that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. While scaling up has been linked to increase unsupervised or at least semi-supervised performance, the same cannot be said in the case of computer vision applications.