Distal Interference: Exploring the Limits of Model-Based Continual Learning