We exemplify our result in two tasks of interest in statistical learning: a) classification for a mixture with sparse means, wherewestudytheefficiencyof `1penaltywithrespectto `2;b)max-marginmulticlass classification, where we characterise the phase transition on the existence ofthemulti-class logistic maximum likelihood estimator forK >2.

Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks. In this contribution, we provide a rigorous justification of these approaches for a two-layers neural network model called the committee machine. We also introduce a version of the approximate message passing (AMP) algorithm for the committee machine that allows to perform optimal learning in polynomial time for a large set of parameters.

Figure 1: Prediction risk of ridge regression on trained CK features (erf) after one feature learning step.

Many high-dimensional statistical inference problems are believed to possess inherentcomputationalhardness.

We consider the problem of estimating a rank-1 signal from a noisy data matrix.

In particular the generative adversarial networks (GANs) [3] provide an impressively powerful waytorepresent classes ofsignals.

Understanding Neural networks ingeneral, and howtotrain Recurrent Neural Networks (RNNs) in particular,remains acentral question in modern machine learning.

One such prior on the low-rank component is sparsity, giving rise to the sparse principal component analysis problem. Unfortunately, there is strong evidence that this problem suffers from a computational-to-statistical gap, which may be fundamental. In this work, we study an alternative prior where the low-rank component is in the range of a trained generative network.

The problem of learning the parameters of a neural network is two-fold. First, we want that their training on a set of data via minimization of a suitable loss function succeed in finding a set of parameters for which the value of the loss is close to its global minimum.

One of the central objects in such algorithms are the so calledstate evolution equations, a low-dimensional recursion equations which allowtoexactly compute the high dimensional distribution of the iterates of the sequence. In this proof we will use a specific form of matrix-valued approximate message-passing iteration with non-separable non-linearities. In its full generality, the validity of the state evolution equations in this case is an extension of the works of [36, 37] includedin[67].