Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum

Following this characterization, we will Wide neural networks are biased towards learning use the term spectral bias of neural networks to refer to the certain functions, influencing both the rate of convergence inductive bias induced by their corresponding NTK spectrum. of gradient descent (GD) and the functions Specifically, it has been observed both theoretically that are reachable with GD in finite training and empirically that for a wide neural network, learning an time. As such, there is a great need for methods eigen-direction of the NTK with GD requires a number of that can modify this bias according to the iterations that is inversely proportional to the corresponding task at hand. To that end, we introduce Modified eigenvalue (Bowman & Montufar, 2022; Fridovich-Keil Spectrum Kernels (MSKs), a novel family of et al., 2021; Xu et al., 2022). Thus, if this spectral bias can constructed kernels that can be used to approximate be modified, it could lead to accelerated network training of kernels with desired eigenvalues for which certain target functions. Typically, the eigenvalue of NTK no closed form is known. We leverage the duality decays at least at a polynomial rate, implying that many between wide neural networks and Neural Tangent eigen-directions cannot be learned in polynomial time with Kernels and propose a preconditioned gradient gradient descent (Ma & Belkin, 2017). As such, modifying descent method, which alters the trajectory the spectral bias of a neural network is necessary to enable of GD. As a result, this allows for a polynomial a feasible learning time, allowing learning target functions and, in some cases, exponential training speedup that are not well aligned with the top eigen-directions of without changing the final solution.