Benchmarking M-LTSF: Frequency and Noise-Based Evaluation of Multivariate Long Time Series Forecasting Models

Abstract--Understanding the robustness of deep learning models for multivariate long-term time series forecasting (ML TSF) remains challenging, as evaluations typically rely on real-world datasets with unknown noise properties. We propose a simulation-based evaluation framework that generates parameterizable synthetic datasets, where each dataset instance corresponds to a different configuration of signal components, noise types, signal-to-noise ratios, and frequency characteristics. These configurable components aim to model real-world multivariate time series data without the ambiguity of unknown noise. This framework enables fine-grained, systematic evaluation of M-L TSF models under controlled and diverse scenarios. Our analysis reveals that all models degrade severely when lookback windows cannot capture complete periods of seasonal patters in the data. S-Mamba and Autoformer perform best on sawtooth patterns, while R-Linear and iTransformer favor sinusoidal signals. White and Brownian noise universally degrade performance with lower signal-to-noise ratio while S-Mamba shows specific trend-noise and iTransformer shows seasonal-noise vulnerability. Further spectral analysis shows that S-Mamba and iTransformer achieve superior frequency reconstruction. This controlled approach, based on our synthetic and principle-driven testbed, offers deeper insights into model-specific strengths and limitations through the aggregation of MSE scores and provides concrete guidance for model selection based on signal characteristics and noise conditions. IME series forecasting plays a crucial role across diverse fields such as energy systems [1]-[3], meteorology [4], [5], traffic flow modeling [6], [7] or the modeling of sensor networks [8], [9]. Reliable forecasts support proactive decision-making, effective risk management, and efficient planning. As high-resolution temporal data becomes increasingly available, the need for robust and scalable forecasting models has grown more important than ever. A time series represents data points ordered in time and can be categorized as either univariate, when consisting of a single variable, or multivariate, when involving multiple interdependent variables [10].