Adversary Is the Best Teacher: Towards Extremely Compact Neural Networks

Why is our contribution important to the community? The recent boom in deep neural networks has resulted in Learning without any explicit supervision for a task ipso their being used for a wide variety of applications, many of facto provides interesting properties to our approach. An example which find significance when run on memory-constrained is that the learning method is domain and task independent, environments. Popular methods for neural network compression since instead of learning a given task, we learn aim to achieve a reduction in the number of parameters a way to learn that from the teacher. Hence, it should be while retaining state-of-the-art results. A seminal work well suited to classification, retrieval, clustering or any other on model compression was by Hinton et al [2] who introduced method across domains. Another interesting fact about this a technique in which a small student network learns approach is that humans learn in a similar way too - they from a large teacher network that is trained to saturation.