Data Unlearning Beyond Uniform Forgetting via Diffusion Time and Frequency Selection

Data unlearning aims to remove the influence of specific training samples from a trained model without requiring full retraining. Unlike concept unlearning, data unlearning in diffusion models remains underexplored and often suffers from quality degradation or incomplete forgetting. To address this, we first observe that most existing methods attempt to unlearn the samples at all diffusion time steps equally, leading to poor-quality generation. We argue that forgetting occurs disproportionately across time and frequency, depending on the model and scenarios. By selectively focusing on specific time-frequency ranges during training, we achieve samples with higher aesthetic quality and lower noise. Finally, to evaluate both deletion and quality of unlearned data samples, we propose a simple normalized version of SSCD. Together, our analysis and methods establish a clearer understanding of the unique challenges in data unlearning for diffusion models, providing practical strategies to improve both evaluation and unlearning performance. The ability to remove the influence of training samples from a learned model, often referred to as machine unlearning (Bourtoule et al., 2021), has become increasingly important. Regulatory frameworks such as the "right to be forgotten" in the General Data Protection Regulation (GDPR) by the European Union and growing concerns about sensitive or proprietary data have created demand for methods that allow models to forget without costly retraining from scratch. Recently, with the development of generative models such as diffusion models (Ho et al., 2020), unlearning the unsafe concept or memorization has been actively explored through training-free sampling (Kim et al., 2025), output filtering (Y oon et al., 2025), and fine-tuning (Wang et al., 2025a).