Learning Dynamic Author Representations with Temporal Language Models

--Language models are at the heart of numerous works, notably in the text mining and information retrieval communities. These statistical models aim at extracting word distributions, from simple unigram models to recurrent approaches with latent variables that capture subtle dependencies in texts. However, those models are learned from word sequences only, and authors' identities, as well as publication dates, are seldom considered. We propose a neural model, based on recurrent language modeling, which aims at capturing language diffusion tendencies in author communities through time. By conditioning language models with author and temporal vector states, we are able to leverage the latent dependencies between the text contexts. This allows us to beat several temporal and non-temporal language baselines on two real-world corpora, and to learn meaningful author representations that vary through time. Index T erms --representation learning, dynamic language model, diachronic text analysis I. I NTRODUCTION Language modeling has been at the heart of a huge amount of works for decades. While the natural language processing field focuses on fine-grained text analysis, statistical models for information retrieval and text mining are essentially based on word (or N-gram) counts, considering more or less complex dependencies in texts. Early works in this area focused on the unigram multinomial model [1], and recent works are shifting toward neural approaches, with distributed representations of words [2], [3]. Research on these deep language models is very active [4]-[8], with applications in various text-related tasks such as speech recognition [9], image captioning [10], or text generation [11]. The goal of the language modeling task is to determine word distributions, depending on their context. Classically, these contexts are limited to previous or surrounding words in text documents. However, textual documents often come with additional contextual information, namely their authors and publication dates. Leveraging this additional contextual information is thus a key challenge in order to build more efficient language models.