Learning from the Undesirable: Robust Adaptation of Language Models without Forgetting