Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer

BRAIN-IT: IMAGE RECONSTRUCTION FROM FMRI VIA BRAIN-INTERACTION TRANSFORMER Roman Beliy, Amit Zalcher, Jonathan Kogman, Navve Wasserman, Michal Irani Department of Computer Science and Applied Mathematics The Weizmann Institute of Science roman.beliy@weizmann.ac.il ABSTRACT Reconstructing images seen by people from their fMRI brain recordings provides a non-invasive window into the human brain. Despite recent progress enabled by diffusion models, current methods often lack faithfulness to the actual seen images. These functional-clusters are shared by all subjects, serving as building blocks for integrating information both within and across brains. All model components are shared by all clusters & subjects, allowing efficient training with a limited amount of data. To guide the image reconstruction, BIT predicts two complementary localized patch-level image features: (i) high-level semantic features which steer the diffusion model toward the correct semantic content of the image; and (ii) low-level structural features which help to initialize the diffusion process with the correct coarse layout of the image. BIT's design enables direct flow of information from brain-voxel clusters to localized image features. Through these principles, our method achieves image reconstructions from fMRI that faithfully reconstruct the seen images, and surpass current SotA approaches both visually and by standard objective metrics. Moreover, with only 1-hour of fMRI data from a new subject, we achieve results comparable to current methods trained on full 40-hour recordings. 1 INTRODUCTION Reconstructing visual experiences from brain activity (fMRI-to-image reconstruction) is a key challenge with broad implications for both neuroscience and brain-computer interfaces (Milekovic, 2018; Naci et al., 2012). Such reconstructions may provide a window into visual perception in the brain, enable the study of visual imagery (Cichy et al., 2012; Pearson et al., 2015), reveal dream content (Horikawa et al., 2013; Horikawa & Kamitani, 2017), and even assist in assessing disorders of consciousness (Monti et al., 2010; Owen et al., 2006). In a typical image decoding setting, subjects view natural images while their brain activity is being recorded using functional Magnetic Resonance Imaging (fMRI). This produces paired data of images and their corresponding fMRI scans. The task is then to reconstruct the perceived image from new (test) fMRI signals. Early work in this domain mapped fMRI signals to handcrafted image features (Kay et al., 2008; Naselaris et al., 2009; Nishimoto et al., 2011), which were then used for image reconstruction. Subsequent studies employed deep learning methods (Beliy et al., 2019; Lin et al., 2019; Shen et al., 2019).