Temporal Dependencies in Feature Importance for Time Series Predictions

Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time. In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues. Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score. We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods, and explore the impact of various evaluation strategies. Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics. Reliably explaining predictions of machine learning models is important given their wide-spread use. Explanations provide transparency and aid reliable decision making, especially in domains such as finance and healthcare, where explainability is often an ethical and legal requirement (Amann et al., 2020; Prenio & Yong, 2021). Multivariate time series data is ubiquitous in these sensitive domains, however explaining time series models has been relatively under explored. In this work we focus on saliency methods, a common approach to explainability that provides explanations by highlighting the importance of input features to model predictions (Baehrens et al., 2010; Mohseni et al., 2020). It has been shown that standard saliency methods underperform on deep learning models used in the time series domain (Ismail et al., 2020). In time series data, observations of the same feature at different points in time are typically related and their order matters.