Improving Unlearning with Model Updates Probably Aligned with Gradients

Machine learning models are integrated into many real-world applications. Since these models contain artifacts of potentially sensitive training data, this raises concerns about data confidentiality and user privacy. The ability to remove specific training data from a model has emerged as a key mechanism to enforce, for instance, the "right to be forgotten" promoted by the European GDPR law [17] or the "right to erase" in the Canadian CPPA legislation [29]. Approximate machine unlearning aims to find efficient mechanisms, avoiding the cost of learning a new model from scratch over the training dataset deprived of the sensitive data. Privacy is not the only application of machine unlearning. It has been proven useful as a defense against backdoor attacks by annihilating the influence of the poisoned training data [45, 38], or to improve fairness by removing data that induce biases in the training set. Another scenario is the derivation of a restricted public model from a powerful private model learned on some sensitive data [12]. The accuracy of the public model should be on par with or slightly degraded compared to the private model.