1000x Smaller GPT-3/2? LoRA: Low-Rank Adaptation of Large Language Models

Is it possible to use large models such as GPT-3 (175B parameters) for downstream tasks with training only 37M parameters and outperform the fine-tuned model? Everyone knows there are lots of problems with the direction Deep Neural Networks models are going. It feels like they are getting larger every minute. While it is beneficial to have these pre-trained models to chose from, it is getting really hard to find enough resources to fine-tune them for any downstream task. But, What if I tell you there is no need to fine-tune the model anymore?