Weight Spectra Induced Efficient Model Adaptation