Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imaging

Transfer Learning Using a light-weight model trained with target dataset directly can outperform the pre-trained TL model using natural images.[30] Self-Supervised Learning Pre-trained SSL model using natural images does not perform well with target COVID-19 samples and need further guidance from user.Problem Summary: domain discrepancy during pre-training willdegrade pre-trained model's performance[31] Transfer Learning Utilising pre-trained TL model does not bring significant improvement tothe target medical dataset with an imbalanced sample distribution.[32] Self-Supervised Learning The imbalanced source and target datasets lead to poor model performance even after self-supervised pre-training.Problem Summary: neither TL or SSL methods show improvedperformance towards imbalanced datasets[33] Transfer Learning The complexity of model and pre-training process makes it hard to understand the results and reduce the reliability of predictions.[34] Self-Supervised Learning The pre-training process of SSL model is fully unsupervised, which raised the concern for whether the model have fully understand the target dataset or is making predictions based on random factors.Problem Summary: the complexity of knowledge transferringprocess raised concerns of model reliabilityTable 1: Four main issues that constrained the application of pre-train methods in the medical field are summarised here: 1. the performance gap between TL and SSL in different data modalities, 2. the domain mismatch gap between source and target domain, 3. the challenge of data imbalance scenarios, 4. the difficulty in model explainability and analysis.