Landscape Complexity for the Empirical Risk of Generalized Linear Models: Discrimination between Structured Data

In its traditional formulation, the activation local minima of cost (or loss) functions of the weights of nonlinear functionσis a step function with a specific threshold, representations (such as perceptrons) of vast amounts of but choosing a smooth activation function brings the model input data. Once the weights are determined they can be used, closer to modern machine learning considerations. As a statistical e.g. for discrimination purposes on datasets stemming from physics model it has many interesting properties such the same distribution. However, understanding the behavior as a jamming and glassy behavior [14]. of these algorithms is obstructed by the unsuitability of traditional We aim to provide qualitative information on these random statistical approaches in dealing with the fact that the loss landscapes by estimating the number of their critical size of the training sample, and the data dimension and weight points about each of their level sets. Specifically we are dimensions are typically quite large and often of the same order interested in the quantity [1, 2]. Furthermore, the performance of local algorithms, like gradient descent (or its variants), depends strongly on the N (B; L)=#{L(w) B| L(w)=0} (2) geometry of the loss landscape they operate on, which is typically highly non-convex [3]. Therefore, the success on modern in the non-trivial scaling limit where d and m/d neural networks would suggest that although their lossfunction β > 1. The corresponding asymptotic annealed critical point might have many critical points (local minima and complexity may be defined as saddle-points), these points are typically rich in generalization.