Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark

Tokenization constitutes an essential preprocessing step within Natural Language Processing (NLP), exerting a direct impact on large language models' (LLMs) capacity to capture syntactic, morphosyntactic, and semantic details. This paper introduces a novel framework for the systematic evaluation of tokenization strategies, with a particular focus on mitigating the challenges associated with morphologically-rich and low-resource languages. Using a Turkish dataset of 6,200 multiple-choice questions derived from the Massive Multitask Language Understanding (MMLU) benchmark, the framework evaluates tokenizers across five key metrics: vocabulary size, token count, processing time, language-specific token percentages (%TR), and token purity. These metrics, proposed in this study, offer a structured approach to assessing how effectively tokenizers preserve linguistic structures. While %TR measures the proportion of valid words generated in the target language, %Pure evaluates the alignment of tokens with meaningful linguistic units, such as roots and valid morphemes, ensuring minimal semantic fragmentation. The findings reveal that language-specific token percentages, introduced as a critical evaluation metric, exhibit a stronger correlation with downstream performance (e.g., MMLU scores) compared to token purity, emphasizing their importance in enhancing model accuracy and robustness. Furthermore, the analysis demonstrates that larger model parameters do not necessarily yield superior tokenization quality or improved results, highlighting the need for tailored tokenization strategies that prioritize linguistic alignment over mere computational scaling. By addressing both computational efficiency and linguistic fidelity, this framework establishes a new standard for developing robust tokenization methods optimized for morphologically complex and low-resource languages. Future work will focus on advancing morphological analysis techniques, exploring domain-specific customizations, and conducting cross-linguistic evaluations to further refine tokenization practices across diverse linguistic contexts.