A beginner's guide to Knowledge Distillation in Deep Learning

By the emergence of deep learning in the large size of data, the true usage of the data has been boosted and deep learning does this by traversing those data on millions of parameters. But this has made an additional requirement of the computation resources such as GPU and these resources are not available at the cutting edge devices such as mobile phones. To counter this issue researchers have introduced many compression techniques such as Knowledge Distillation which is the process of converting complex model behaviour to smaller one in terms of a number of parameters. So in this article, we will take a look at knowledge distillation and will discuss its context briefly. Below are the major points listed that are to be discussed in this article.