Transformers in Time-series Analysis: A Tutorial

Transformers belong to a class of machine learning models that use self-attention or the scaled dot-product operation as their primary learning mechanism. Transformers were initially proposed for neural machine translation - one of the most challenging natural language processing (NLP) tasks [1]. Recently, Transformers have been successfully employed to tackle various problems in machine learning and achieve state-of-the-art performance [2]. Apart from classical NLP tasks, examples from other areas include image classification [3], object detection and segmentation [4], image and language generation [5], sequential decision-making in reinforcement learning [6], multi-modal (text, speech, and image) data processing [7], and analysis of tabular and time-series data [8]. This tutorial paper focuses on time-series analysis using Transformers. Time-series data consist of ordered samples, observations, or features recorded sequentially over time. Time-series datasets often arise naturally in many real-world applications where data is recorded over a fixed sampling interval. Examples include stock prices, digitized speech signals, traffic measurements, sensor data for weather patterns, biomedical measurements, and various kinds of population data recorded over time.