Learning and Forgetting Unsafe Examples in Large Language Models