Vocabulary Transfer for Medical Texts

Vocabulary transfer is a transfer learning subtask in which language models fine-tune with the corpus-specific tokenization instead of the default one, which is being used during pretraining. This usually improves the resulting performance of the model, and in the paper, we demonstrate that vocabulary transfer is especially beneficial for medical text processing. Using three different medical natural language processing datasets, we show vocabulary transfer to provide up to ten extra percentage points for the downstream classifier accuracy.