We study the optimization problem associated with fitting two-layer ReLU neural networks with respect to the squared loss, where labels are generated by a target network. We make use of the rich symmetry structure to develop a novel set of tools for studying families of spurious minima. In contrast to existing approaches which operate in limiting regimes, our technique directly addresses the nonconvex loss landscape for a finite number of inputs d and neurons k, and provides analytic, rather than heuristic, information.

Nonlinear independent component analysis (ICA) is fundamental in unsupervised learning.

Submodular maximization has become established as the method of choice for the task of selecting representative anddiversesummaries of data.

However, known lower bounds ignore the problem's geometry,including itscondition number,whereas theupper bounds explicitly depend on it. Perhaps surprisingly, we prove that when the condition number is taken into account, without-replacement SGDdoesnotsignificantly improveon withreplacement SGD in terms of worst-case bounds, unless the number of epochs (passes overthedata) islargerthanthecondition number.