Oracle Complexity in Nonsmooth Nonconvex Optimization

It is well-known that given a smooth, bounded-from-below, and possibly nonconvex function, standard gradient-based methods can find \epsilon -stationary points (with gradient norm less than \epsilon) in \mathcal{O}(1/\epsilon 2) iterations. However, many important nonconvex optimization problems, such as those associated with training modern neural networks, are inherently not smooth, making these results inapplicable. In this paper, we study nonsmooth nonconvex optimization from an oracle complexity viewpoint, where the algorithm is assumed to be given access only to local information about the function at various points. We provide two main results (under mild assumptions): First, we consider the problem of getting \emph{near} \epsilon -stationary points. This is perhaps the most natural relaxation of \emph{finding} \epsilon -stationary points, which is impossible in the nonsmooth nonconvex case.