TimelyGPT: Recurrent Convolutional Transformer for Long Time-series Representation

Pre-trained models (PTMs) have gained prominence in Natural Language Processing and Computer Vision domains. When it comes to time-series PTMs, their development has been limited. Previous research on time-series transformers has mainly been devoted to small-scale tasks, yet these models have not consistently outperformed traditional models. Additionally, the performance of these transformers on large-scale data remains unexplored. These findings raise doubts about Transformer's capabilities to scale up and capture temporal dependencies. In this study, we re-examine time-series transformers and identify the shortcomings of prior studies. Drawing from these insights, we then introduce a pioneering architecture called Timely Generative Pre-trained Transformer (TimelyGPT). This architecture integrates recurrent attention and temporal convolution modules to effectively capture global-local temporal dependencies in long sequences. The relative position embedding with time decay can effectively deal with trend and periodic patterns from time-series. Our experiments show that TimelyGPT excels in modeling continuously monitored biosignal as well as irregularly-sampled timeseries data commonly observed in longitudinal electronic health records. This breakthrough suggests a priority shift in time-series deep learning research, moving from small-scale modeling from scratch to large-scale pre-training. Time-series data mining holds significant importance in healthcare, given its potential to trace patient health trajectories and predict medical outcomes (Ma et al., 2023b; Eldele et al., 2021; Fawaz et al., 2019). In the field of healthcare, there are two primary categories: continuous and irregularlysampled time-series data. Continuous time-series, such as biosignals, has been extensively studied in various applications including health monitoring (Stirling et al., 2020), disease classification (Phan et al., 2021), and physical activity prediction (Reiss et al., 2019b). Irregularly-sampled time series is commonly found in clinical records, where spontaneous updates are made to an individual patient's health status (Zhang et al., 2022b). The key challenge is to extract meaningful representation from these time-series, especially when there is limited labeled data available. A promising approach to overcome this constraint is to adopt transfer learning (Ma et al., 2023b). Initially, a model is pretrained on large datasets to capture temporal representation.