Spectral Adapter: Fine-Tuning in Spectral Space