Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. Further, the standard fMRI encoding procedure does not account for varying temporal properties in the encoding features.

Additional subject flatmaps are shown in figures 2-7 at the end of the document. Only significantly predicted voxels are shown. These flatmaps correspond to figures 3-5 in the main text and follow the same colormap. Note that subject S04 is excluded from this study due to poor data quality, resulting in 6 subjects overall. Results from subject S03 (highest number of significant voxels) are shown in the main text.

Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. Further, the standard fMRI encoding procedure does not account for varying temporal properties in the encoding features.

Summary and Contributions: Update after rebuttal: In my original review, I asked for a better motivation for the voxel timescale estimate provided by the authors in Eq. 6. R1 also appears to have concerns about this timescale estimation. As it turns out, there was a typo in Eq. 6 (pointed out in the rebuttal). However, the authors do not offer more motivation about using this specific estimate of the voxel timescale. I am glad they provided a visualization in Figure 1B in the supplementary (which should be referenced in the main paper) that is similar to what I suggested but it's not clear to me that the visualization actually shows what the authors conclude. For example, it's not clear whether the two plots have the same scale of beta magnitudes.

The reviewers appreciated this paper's approach to explaining voxel activation as a function of time-lagged stimuli features. There were a few mistakes in the paper that were uncovered in the reviews, but adequately addressed in the author's rebuttal. The reviewers point out that the authors used a less-accurate LSTM model when more accurate models exist (e.g. In general the reviewers were positive about this usage of a variant LSTM to explain the temporal nature of language processing in the brain.

Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. Further, the standard fMRI encoding procedure does not account for varying temporal properties in the encoding features.