Appendix Organization of Appendix. In Appendix A we present details of the numerical experiments and

We give more discussions of our work in Appendix B. Appendix C provides missing technical details of Section 3 while Appendix D provides those of Section 4. In A.1, we present the details of the numerical experiments. A.4, we conduct experiments when the balanced initialization condition is not satisfied. We conduct over-parameterized regression with different linear networks. We now present the details of numerical experiments conducted in Section 5. ` GD and SGD when k (0) k is extremely large. Other settings are similar to that of rank-1 linear networks.