Medical image segmentation is a crucial task in medical image processing. Thanks to the advent of CNN[12], U-Net [17], and their variants such as V-Net[14], 3D U-Net[5], Res-UNet[15], Dense-UNet[13], we are able to perform segmentation task with precision. More recently, with implementations of transformer-based models, the medical imaging community enjoyed satisfying success in segmentation tasks. Networks like Medical Transformers[18] and SwinUnet[1] push the front-line boundary to another degree. Others have implemented learning methodologies from other fields, such as dictionary learning, to work on medical images. KEN[16] - knowledge embedding network - for example, takes advantage of the fruitfulness of information embedding in each layer via dictionary learning to provide a more semantically meaningful network.

Transfer learning has gained attention in medical image analysis due to limited annotated 3D medical datasets for training data-driven deep learning models in the real world. Existing 3D-based methods have transferred the pre-trained models to downstream tasks, which achieved promising results with only a small number of training samples. However, they demand a massive amount of parameters to train the model for 3D medical imaging. In this work, we propose a novel transfer learning framework, called Medical Transformer, that effectively models 3D volumetric images in the form of a sequence of 2D image slices. To make a high-level representation in 3D-form empowering spatial relations better, we take a multi-view approach that leverages plenty of information from the three planes of 3D volume, while providing parameter-efficient training. For building a source model generally applicable to various tasks, we pre-train the model in a self-supervised learning manner for masked encoding vector prediction as a proxy task, using a large-scale normal, healthy brain magnetic resonance imaging (MRI) dataset. Our pre-trained model is evaluated on three downstream tasks: (i) brain disease diagnosis, (ii) brain age prediction, and (iii) brain tumor segmentation, which are actively studied in brain MRI research. The experimental results show that our Medical Transformer outperforms the state-of-the-art transfer learning methods, efficiently reducing the number of parameters up to about 92% for classification and