NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time Series Pretraining

Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into nonoverlapping windows. Each window is then characterized by its normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical scales to highdimensional vectors, we propose a numerically multi-scaled embedding module enumerating all possible scales for the scalar values. The model undergoes pretraining using the proposed numerically multi-scaled embedding with a simple contrastive objective on a large-scale dataset containing over a million sequences. We study its transfer performance on a number of univariate and multivariate classification benchmarks. Our method exhibits remarkable improvement against previous representation learning approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods. Despite the phenomenal achievement of large-scale representation learning on various data modalities (Brown et al., 2020; Radford et al., 2021; Caron et al., 2021), the research for time-series representation learning is mostly limited to small-scale datasets without attaining generalization capabilities (Eldele et al., 2021b; Yue et al., 2022; Zhang et al., 2022). Since time-series data may cover a diverse range of domains, such as medical, weather, traffic and more, large-scale training across domains brings special challenges and opportunities for transfer learning. We notice a unique characteristic of time-series data and its representation.