Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language

Tokenization is an important part of the pre-processing step for training and fine-tuning Large Language Models [8]. Thus, the performance of the models also depends on the performance of its tokenizers [8, 5]. Typically, Transformer-based LLMs employ tokenization methods such as WordPiece or Byte Pair Encoding (BPE) [9, 10]. In the WordPiece method, the primary operation involves counting and merging the most frequent subword pairs. The frequency of a subword pair (X Y) can be conceptually represented as [21]: Count(X Y) = number of occurrences of the subword pair (X, Y) in the corpus (1) Whereas, in Byte Pair Encoding (BPE) the main operation is to count and merge the most frequent adjacent symbol pairs, and the frequency of an adjacent symbol pair (a, b) can be conceptually determined by [10]: Count(ab) = number of occurrences of the pair (a, b) in the corpus (2) Byte Pair Encoding (BPE), originally introduced by Sennrich et al. [21] for NLP tasks, is a tokenization algorithm that learns from subword-based encoding from training data. It follows a bottom-up approach where the training dataset is divided into individual characters or tokens, which are then aggregated together in pairs of tokens based on the number of occurrences. BPE has become a near-universal choice for modern language models [10, 21].