Sleep plays an extremely important role in human health. Ensuring an adequate amount of high-quality sleep is essential for maintaining physical health and psychological balance. Professional measurement of sleep state is mainly conducted through Polysomnography (PSG) [1]. However, PSG involves a significant physical burden on the subjects and is difficult to measure without specialized facilities or hospitals. In recent years, evaluation methods utilizing wearable devices have been developed with the aim of facilitating sleep assessment [2]. However, the information that can be obtained through a smartwatch is limited, typically encompassing data such as acceleration and heart rate. While EEG-based sleep monitoring offers high accuracy, the requirement to wear headgear, even for a single-channel EEG headset [3], presents a significant burden.

Recurrent neural networks are a standard building block in numerous machine learning domains, from natural language processing to time-series classification. While their application has grown ubiquitous, understanding of their inner workings is still lacking. In practice, the complex decision-making in these models is seen as a black-box, creating a tension between accuracy and interpretability. Moreover, the ability to understand the reasoning process of a model is important in order to debug it and, even more so, to build trust in its decisions. Although considerable research effort has been guided towards explaining black-box models in recent years, recurrent models have received relatively little attention. Any method that aims to explain decisions from a sequence of instances should assess, not only feature importance, but also event importance, an ability that is missing from state-of-the-art explainers. In this work, we contribute to filling these gaps by presenting TimeSHAP, a model-agnostic recurrent explainer that leverages KernelSHAP's sound theoretical footing and strong empirical results. As the input sequence may be arbitrarily long, we further propose a pruning method that is shown to dramatically improve its efficiency in practice.