Differentially Private Learning with Adaptive Clipping

There have been a lot of recent advances in iterative training methods like stochastic gradient descent (SGD), one of the main reasons being thier applicability in training neural networks. Deep learning has found a variety of applications, including image classification, language translation, and music generation [25, 12, 26, 4]. To effectively perform such tasks, neural networks need a large amount of data for training. It is often the case that these datasets contain a lot of sensitive information. Moreover, many recent works [10, 27, 24, 5, 20] have shown that it is possible to extract sensitive information about the training data just from the parameters of a trained model. Thus, it becomes imperative to use learning techniques that provide a rigorous guarantee of privacy for the training data used. Differential privacy (DP) [8, 9] has been recently used as a gold standard to bound the privacy leakage of sensitive data when performing learning tasks. Intuitively, DP prevents an adversary from confidently making any conclusions about whether a sample was used in training a model, even while having access to arbitrary side information.