Vocabulary embeddings organize linguistic structure early in language model training

Here, we ask: how are the input vocabulary representations of language models structured, and how and when does this structure evolve over training? To answer this question, we use representational similarity analysis, running a suite of experiments that correlate the geometric structure of the input embeddings and output embeddings of two open-source models (Pythia 12B and OLMo 7B) with semantic, syntactic, and frequency-based metrics over the course of training. Our key findings are as follows: 1) During training, the vocabulary embedding geometry quickly converges to high correlations with a suite of semantic and syntactic features; 2) Embeddings of high-frequency and function words (e.g., "the," "of") converge to their final vectors faster than lexical and low-frequency words, which retain some alignment with the bias in their random initializations. These findings help map the dynamic trajectory by which input embeddings organize around linguistic structure, revealing distinct roles for word frequency and function. Our findings motivate a deeper study of how the evolution of vocabulary geometry may facilitate specific capability gains during model training. Token embeddings are the input vectors to transformer language models. The information that differentiates one input from another, and spurs the diverse and complex processing in large language models, all originates in the vector space of the token embeddings. Understanding the structure of vocabulary embedding representation is therefore a fundamental step in the effort to trace and interpret the internal mechanisms of language models. In this paper, we analyze the representational space of the token embeddings of 153 Pythia 12-billion checkpoints (Biderman et al., 2023) and 186 OLMo 7-billion checkpoints (Groeneveld et al., 2024), and analyze how the representational relationships in the vocabulary matrix form over the course of training.