Learning to Refuse: Towards Mitigating Privacy Risks in LLMs