BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training

Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstream performance. We introduce Picky BPE, a modified BPE algorithm that carries Figure 1: An example of a series of merges to produce a out vocabulary refinement during tokenizer token Kentucky. The pre-merge token frequencies are training. Our method improves vocabulary efficiency, demonstrated in corresponding circles. In the vanilla eliminates under-trained tokens, and BPE algorithm, entucky should also be stored in the does not compromise text compression. Our vocabulary, whereas it is redundant after the merge. In experiments show that our method does not this example, the IoS metric effectively captures the reduce the downstream performance, and in intermediate token, as IoS(entucky) T = 0.9.