Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion