Erasing Without Remembering: Safeguarding Knowledge Forgetting in Large Language Models