Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective