In the past year, the field of Natural Language Processing (NLP) has seen the rise of pretrained language models such as as ELMo (Peters et al., 2018), ULMFiT (Howard and Ruder, 2018) and BERT (Devlin et al., 2019). These approaches train a deep - learning language model on large volumes of unlabeled text, which is subsequently fine - tuned for particular NLP tasks. Applying these models to th e General Language Understanding Evaluation (GLUE) benchmark introduced by Wang et al. (2018) has achieved the best performance to date on tasks ranging from sentiment classification to question answering (Devlin et al., 2019). The benefit of these models has also been demonstrated in specialized NLP domains. BioBERT (Lee et al., 2019), a version of BER T trained exclusively on biomedical text, was able to significantly increase performance on biomedical named entity recognition. Further refining this model on clinical text produced an increase in performance in medical natural language inference (Alsentz er et al. 2019). While large pretrained models offer significantly increased performance, they come with their own constraints, as the number of parameters in the classic BERT - base model exceeds 100 million. As such, their computational cost can thus be p rohibitively high at both training and prediction time (Devlin et al., 2019). More recent work has addressed this challenge by'distilling' the models, training smaller versions of BERT which reduce the number of parameters to train by 40% while retaining more than 95% of the full model performance and even outperforming it on two out of eleven GLUE tasks (Sanh et al., 2019).