On the Growth of Mistakes in Differentially Private Online Learning: A Lower Bound Perspective

With the increasing need to protect the privacy of sensitive user data while conducting meaningful data analysis, Differential Privacy (DP) [14] has become a popular solution. DP algorithms ensure that the impact of any single data sample on the output is limited, thus safeguarding individual privacy. Several works have obtained DP learning algorithms for various learning problems in both theory and practice. However, privacy does not come for free and often leads to a statistical (and sometimes computational) cost. The classical solution for non-private Probably Approximately Correct (PAC) learning [28] is via Empirical Risk Minimisation (ERM) that computes the best solution on the training data. Several works [3, 11] have shown that incorporating DP into ERM incurs a compulsory statistical cost that depends on the dimension of the problem. In the well-known setting of PAC learning with DP, Kasiviswanathan et al. [23] provided the first guarantees that all finite VC classes can be learned with a sample size that grows logarithmically in the size of the class. This line of research was advanced by subsequent works [4, 6, 17], resulting in the findings of Alon et al. [2] which established a surprising equivalence between non-private online learning and Approximate DP-PAC learning.