How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives

Open in new window