Psychiatric evaluation nowadays is heavily dependent on the patient's verbal report about disturbed feelings, thoughts, behaviors and their changes over time. Accordingly, evaluation hinges on two main components: unstructured interviews, which allow patients to express themselves freely under the guidance of open questions, and structured questionnaires, aimed at standardizing the assessment. These methods are outlined in the Diagnostic and Statistical Manual of Mental Disorders (DSM) series, which attempts to assign universal scores to individual experiences of mental disorders [1]. However, the inherent complexity of mental health conditions, characterized by a known positive manifold of symptoms and compounded by the subjective nature and potential unreliability of self-reported data (especially from one session to another), along with interviewer biases, make accurate diagnosis challenging. The overlapping symptoms and the instability of mental state, especially in pathological conditions, further complicate the need for precision, precluding an objective and quantitative account of a critical element in the psychiatric evaluation process; the subjective self-experience [2, 3, 4]. The evolution of psychiatric practice is increasingly shaped by the integration of Natural Language Processing (NLP) and machine learning within traditional diagnostic approaches.

We present TFF, which is a Transformer framework for the analysis of functional Magnetic Resonance Imaging (fMRI) data. TFF employs a two-phase training approach. First, self-supervised training is applied to a collection of fMRI scans, where the model is trained to reconstruct 3D volume data. Second, the pre-trained model is fine-tuned on specific tasks, utilizing ground truth labels. Our results show state-of-the-art performance on a variety of fMRI tasks, including age and gender prediction, as well as schizophrenia recognition. Our code for the training, network architecture, and results is attached as supplementary material.

Deep neural networks have been developed drawing inspiration from the brain visual pathway, implementing an end-to-end approach: from image data to video object classes. However building an fMRI decoder with the typical structure of Convolutional Neural Network (CNN), i.e. learning multiple level of representations, seems impractical due to lack of brain data. As a possible solution, this work presents the first hybrid fMRI and deep features decoding approach: collected fMRI and deep learnt representations of video object classes are linked together by means of Kernel Canonical Correlation Analysis. In decoding, this allows exploiting the discriminatory power of CNN by relating the fMRI representation to the last layer of CNN (fc7). We show the effectiveness of embedding fMRI data onto a subspace related to deep features in distinguishing semantic visual categories based solely on brain imaging data.