BiTimeBERT: Extending Pre-Trained Language Representations with Bi-Temporal Information

Time is an important aspect of documents and is used in a range of Temporal signals constitute significant features in various types NLP and IR tasks. In this work, we investigate methods for incorporating of text documents such as news articles or biographies. They can temporal information during pre-training to further improve be leveraged to understand chronology, causalities, developments, the performance on time-related tasks. Compared with common and ramifications of events, being helpful in a range of different pre-trained language models like BERT which utilize synchronic NLP tasks. Utilizing temporal signals in information retrieval has received document collections (e.g., BookCorpus and Wikipedia) as the training considerable attention recently, too. For example, researchers corpora, we use long-span temporal news article collection for have addressed time-sensitive queries in search leading to the formation building word representations. We introduce BiTimeBERT, a novel of a subset of Information Retrieval called Temporal Information language representation model trained on a temporal collection Retrieval [8, 26] in which both query and document of news articles via two new pre-training tasks, which harnesses temporal aspects are of key concern. Event detection and ordering two distinct temporal signals to construct time-aware language [14, 47], timeline summarization [2, 10, 36, 46, 50], event occurrence representations. The experimental results show that BiTimeBERT time prediction [54], temporal clustering [9], question answering consistently outperforms BERT and other existing pre-trained models [39, 52] and semantic change detection [41, 42] are other example with substantial gains on different downstream NLP tasks and tasks where utilizing temporal information has proven beneficial.