TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction

Magnetic resonance imaging (MRI) and Positron emission tomography (PET) could help more accurately predict MCI The prediction of mild cognitive impairment (MCI) conversion conversion [2]. to Alzheimer's disease (AD) is important for early Convolutional neural networks (CNNs) have been widely treatment to prevent or slow the progression of AD. To accurately applied to AD classification and prediction from imaging predict the MCI conversion to stable MCI or progressive data. Valliani et al. [3] fine-tuned a pretrained ResNet-50 MCI, we propose TriFormer, a novel transformer-based to classify AD and CN based on 2D axial slices. Wen et framework with three specialized transformers to incorporate al. [4] leveraged 3D spatial information by using a 3D CNN multi-modal data. TriFormer uses I) an image transformer to and outperformed previous 2D-based methods in AD classification extract multi-view image features from medical scans, II) a and MCI conversion prediction. However, both clinical transformer to embed and correlate multi-modal clinical 2D and 3D CNNs have a strong inductive bias towards local data, and III) a modality fusion transformer that produces receptive fields, which could limit the performance on an accurate prediction based on fusing the outputs from the high dimensional data [5]. Recently, transformers have been image and clinical transformers. Triformer is evaluated on the shown to be effective in capturing global long-range dependency Alzheimer's Disease Neuroimaging Initiative (ADNI) 1 and within imaging [6] and sequential data [7]. They also ADNI2 datasets and outperforms previous state-of-the-art have no indictive bias compared with CNNs.