Toward a Theory of Tokenization in LLMs

Open in new window