Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning