Fine-Tuning Transformers: Vocabulary Transfer

The transformer gave rise to such models as GPT [2, 3] or BERT [4]. These architectures are shown to beat the state of the art for various Natural Language Processing tasks. The performance of such models improves with the size and training of such architectures from scratch requires a lot of computational power and huge datasets. These obstacles hinder the broader adoption of these architectures and limit the majority of successful applications to transfer learning: a huge pretrained model is fine-tuned on a smaller dataset collected for a specific downstream task. This stimulates a growing interest to transfer learning procedures and gives rise to various approaches and practices aimed to raise the effectiveness of the transfer.