A key question for adapting modern deep learning architectures to functional MRI (fMRI) is how to represent the data for model input. To bridge the modality gap between fMRI and natural images, we transform the 4D volumetric fMRI data into videos of 2D fMRI activity flat maps. We train Vision Transformers on 2.3K hours of fMRI flat map videos from the Human Connectome Project using the spatiotemporal masked autoencoder (MAE) framework. We observe that masked fMRI modeling performance improves with dataset size according to a strict power scaling law. Downstream classification benchmarks show that our model learns rich representations supporting both fine-grained state decoding across subjects, as well as subject-specific trait decoding across changes in brain state. This work is part of an ongoing open science project to build foundation models for fMRI data. Our code and datasets are available at https://github.com/MedARC-AI/fmri-fm.

This paper compares the embedding of a 3-layer LSTM to the neural responses of people listening to podcasts recorded via fMRI. The experiments vary the number of layers in the LSTM, and then context available to the LSTM and compare it to a context-free word embedding model. This is a strong paper, well written and clear. The results are thorough and there are a few interesting surprises. I have a few questions of clarification. 1) How do the authors account for the differences in number of words per TR due to differing word length and prosody?