We theoretically study the landscape of the training error for neural networks in overparameterized cases.

An overarching goal in machine learning is to build ageneralizable model with fewsamples.

We then integrate these findings toobtain anoverallperformance guarantee forourmetalearning algorithm.

Recently there has been significant theoretical progress on understanding the convergence andgeneralization ofgradient-based methods onnonconvexlosses withoverparameterized models. Nevertheless, manyaspectsofoptimization and generalization and in particular the critical role of small random initialization are not fully understood.

Onone hand, it is widely accepted that, for two-layer neural networks, the number of parameters should growlinearlywith n(e.g.,[21,38]).

How does overparameterization (e.g., increasing width and depth) of neural networks affect the