few-shot segmentation require few-parameter fine-tuning
Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning
Freezing the pre-trained backbone has become a standard paradigm to avoid overfitting in few-shot segmentation. In this paper, we rethink the paradigm and explore a new regime: {\em fine-tuning a small part of parameters in the backbone}. We present a solution to overcome the overfitting problem, leading to better model generalization on learning novel classes. Our method decomposes backbone parameters into three successive matrices via the Singular Value Decomposition (SVD), then {\em only fine-tunes the singular values} and keeps others frozen. The above design allows the model to adjust feature representations on novel classes while maintaining semantic clues within the pre-trained backbone. We evaluate our {\em Singular Value Fine-tuning (SVF)} approach on various few-shot segmentation methods with different backbones. We achieve state-of-the-art results on both Pascal-5$^i$ and COCO-20$^i$ across 1-shot and 5-shot settings. Hopefully, this simple baseline will encourage researchers to rethink the role of backbone fine-tuning in few-shot settings.
Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning - Supplementary Material Y anpeng Sun 1, Qiang Chen
Therefore, we compared the impact of different training strategies on benchmark datasets. Method fine-tune method Fold-0 Fold-1 Fold-2 Fold-3 Mean baseline freeze backbone 65.60 70.28 64.12 60.27 65.07 SVF 67.42 71.57 In Table 2, we compared the effect of this trick on FSS models. Flod-2, the trend of improvement is very obvious. Y ang(12) was tested with 4000 images, and CyCTR (15) was tested with 5000 images.
Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning
Freezing the pre-trained backbone has become a standard paradigm to avoid overfitting in few-shot segmentation. In this paper, we rethink the paradigm and explore a new regime: {\em fine-tuning a small part of parameters in the backbone}. We present a solution to overcome the overfitting problem, leading to better model generalization on learning novel classes. Our method decomposes backbone parameters into three successive matrices via the Singular Value Decomposition (SVD), then {\em only fine-tunes the singular values} and keeps others frozen. The above design allows the model to adjust feature representations on novel classes while maintaining semantic clues within the pre-trained backbone.