A Simple Baseline for Stable and Plastic Neural Networks