Dynamic Privacy Budget Allocation Improves Data Efficiency of Differentially Private Gradient Descent

In response to the growing demand, differential-private (DP) machine learning [10] provides a computational framework for privacy protection and has been widely studied in various settings, including both convex and non-convex optimization [15, 33, 34]. One widely used procedure for privacy-preserving learning is the (Differentially) Private Gradient Descent (PGD) [1, 3]. A typical gradient descent procedure updates its model by gradients of the loss evaluated on a training dataset. When the data is sensitive, the gradients should be privatized to prevent excess privacy leakage. The PGD privatizes a gradient by adding controlled noise. As such, the models from PGD is expected to have a lower utility as compared to those from unprotected algorithms. In the cases where strict privacy control is exercised, or equivalently, a tight privacy budget, accumulating effects from highly-noised gradients may lead to unacceptable model performance. It is thus critical to design effective privatization procedures for PGD to maintain a great balance between utility and privacy. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.