Continual Learning via Sparse Memory Finetuning