Decoding Natural Images from EEG for Object Recognition

Electroencephalography (EEG) signals, known for the convenient non-invasive acquisition but low signal-to-noise, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. With the framework, we attain significantly above-chance results on a comprehensive EEG-image dataset, achieving a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in challenging 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plausibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios. Our daily life relies on accurately and rapidly identifying objects in complex visual environments (DiCarlo & Cox, 2007). Researchers have pursued decoding natural images from brain activity, aiming to deepen our understanding of the brain and create user-friendly brain-computer interfaces (BCIs) (Kamitani & Tong, 2005; Kay et al., 2008; Gao et al., 2021). Functional magnetic resonance imaging (fMRI), which records blood oxygen level-dependent signals, has been a popular choice for categorizing objects observed by humans (Du et al., 2023; Horikawa & Kamitani, 2017; Allen et al., 2022). Despite its high spatial resolution, fMRI typically requires several seconds to provide a stable response to a single stimulus, limiting its real-time applicability in daily interactions (Lin et al., 2022). Similarly, Magnetoencephalogram (MEG), with high time resolution, has been employed for this purpose, but it is hindered by cost and large devices (Cichy et al., 2014; Hebart et al., 2023). Electroencephalogram (EEG) has emerged as a valuable tool for decoding images based on visualevoked brain activities (Spampinato et al., 2017). EEG has high time resolution, low cost, and good portability, but the low signal-to-noise ratio takes problems (Pan et al., 2022; Kobler et al., 2022).