One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning