Transformer to CNN: Label-scarce distillation for efficient text classification

Significant advances have been made in Natural Language Proc essing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate results, even when there is little labelled data, because these NLP mo dels can benefit from training on both task-agnostic and task-specific unlabelle d data. However, these advantages come with significant size and computational cos ts. This workshop paper outlines how our proposed convolutiona l student architecture, having been trained by a distillation process from a la rge-scale model, can achieve 300 inference speedup and 39 reduction in parameter count. In some cases, the student model performance surpasses its teacher on the studied tasks.