Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel

Recent works have shown that on sufficiently over-parametrized neural nets, gradient descent with relatively large initialization optimizes a prediction function in the RKHS of the Neural Tangent Kernel (NTK).