The Elusive Pursuit of Replicating PATE-GAN: Benchmarking, Auditing, Debugging

Privacy-preserving synthetic data has been increasingly adopted to share data within and across organizations while reducing privacy risks. The intuition is to train a generative model on the real data, draw samples from the model, and create new (synthetic) data points. As the original data may contain sensitive and/or personal information, synthetic data can be vulnerable to membership/property inference, reconstruction attacks, etc. [6, 13, 25, 29, 30, 57]. Thus, models should be trained to satisfy robust definitions like Differential Privacy (DP) [19, 20], which bounds the privacy leakage from the synthetic data. Combining generative models with DP has been advocated for or deployed by government agencies [2, 31, 46, 62], regulatory bodies [60, 61], and non-profit organizations [48, 63].