The Test of Tests: A Framework For Differentially Private Hypothesis Testing

Hypothesis tests are one of the most basic and common statistical analyses that analysts perform on data. The goal of a hypothesis test is to see whether some "effect" in the data (e.g., men are taller than women) is plausibly the result of random variation in the sample, rather than a true fact about the population. Hypothesis tests are the bedrock of statistical analysis in the social sciences, medicine, and other fields, and a variety of hypothesis tests are used, depending on the type of data and the sort of effect one is considering. However, data in these fields often consists of private information about individuals. Researchers are under moral and legal obligations to protect the privacy of that data and can often only access that data if they can guarantee their analysis will not violate the privacy of those individuals. Differential privacy has emerged as the most convincing formal definition of privacy protection in this setting.