Differentially Private Optimization for Smooth Nonconvex ERM

Privacy protection has become a central issue in machine learning algorithms, and differential privacy [Dwork and Roth, 2014] is a rigorous and popular framework for quantifying privacy. We propose a differentially private optimization algorithm that finds an approximate second-order solution for (possibly nonconvex) ERM problems. We propose several techniques to improve the practical performance of the method, including backtracking line search, mini-batching, and a heuristic to avoid the effects of conservative assumptions made in the analysis.