Knowledge Distillation, aka Teacher-Student Model

With the release of large models in the last few years, from GPT-3 to Megatron, I keep pondering how to experiment and use these models for a specific use case. These models are trained on massive corpuses of data (100 GBs) with billions of parameters. Training or performing inference using such a model requires heavy computing and cost. To overcome the above challenges, there are techniques developed to transfer the knowledge learnt by a larger model (teacher) and instil it into a smaller model (student). Here the knowledge refers to parameters learnt during model training.