Real-world business applications require a trade-off between language model performance and size. We propose a new method for model compression that relies on vocabulary transfer. We evaluate the method on various vertical domains and downstream tasks. Our results indicate that vocabulary transfer can be effectively used in combination with other compression techniques, yielding a significant reduction in model size and inference time while marginally compromising on performance.

Vocabulary transfer is a transfer learning subtask in which language models fine-tune with the corpus-specific tokenization instead of the default one, which is being used during pretraining. This usually improves the resulting performance of the model, and in the paper, we demonstrate that vocabulary transfer is especially beneficial for medical text processing. Using three different medical natural language processing datasets, we show vocabulary transfer to provide up to ten extra percentage points for the downstream classifier accuracy.

The transformer gave rise to such models as GPT [2, 3] or BERT [4]. These architectures are shown to beat the state of the art for various Natural Language Processing tasks. The performance of such models improves with the size and training of such architectures from scratch requires a lot of computational power and huge datasets. These obstacles hinder the broader adoption of these architectures and limit the majority of successful applications to transfer learning: a huge pretrained model is fine-tuned on a smaller dataset collected for a specific downstream task. This stimulates a growing interest to transfer learning procedures and gives rise to various approaches and practices aimed to raise the effectiveness of the transfer.