SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models