A Convergence on Two-Layer Nonlinear Networks We consider the family of neural networks f (x) = 1 p p null

Lemma A.2. Assume W (0), β (0) and b have i.i.d. The proof for (A.5) is similar since V ar( To prove (A.6), since | y With a union bound argument, we can show (A.6). Finally, (A.7) followed from standard Gaussian tail bounds and union bound argument, yielding P(max Under the conditions of Theorem 3.2, we define matrices G(0),H (0) R Under the conditions of Theorem 3.2, if the error bound (3.1) holds for all t = 1, 2,...,t From the feedback alignment updates (A.3), we have for all t T | β Lemma A.5. Assume all the inequalities from Lemma A.2 hold. Under the conditions of Theorem 3.2, if the bound for the weights difference (3.2) holds for all t t We prove the inequality (3.1) by induction. Suppose (3.1) and (3.2) hold for all t = 1, 2,...,t Assume all the inequalities from Lemma A.2 hold.