Jointly Exploring Client Drift and Catastrophic Forgetting in Dynamic Learning