Lifelong Learning with Task-Specific Adaptation: Addressing the Stability-Plasticity Dilemma