Unleashing the Power of Randomization in Auditing Differentially Private ML

Differential privacy (DP), introduced in [21], has gained widespread adoption by governments, companies, and researchers by formally ensuring plausible deniability for participating individuals. This is achieved by guaranteeing that a curious observer of the output of a query cannot be confident in their answer to the following binary hypothesis test: did a particular individual participate in the dataset or not? For example, introducing sufficient randomness when training a model on a certain dataset ensures a desired level of differential privacy. This in turn ensures that an individual's sensitive information cannot be inferred from the trained model with high confidence. However, calibrating the right amount of noise can be a challenging process. It is easy to make mistakes when implementing a DP mechanism as it can involve intricacies like micro-batching, sensitivity analysis, and privacy accounting. Even with a correct implementation, there are several known incidents of published DP algorithms with miscalculated privacy guarantees that falsely report higher levels of privacy [16, 33, 39, 46, 56, 57]. Data-driven approaches to auditing a mechanism for a violation of a claimed privacy guarantee can significantly mitigate the danger of unintentionally leaking sensitive data.