b7ae8fecf15b8b6c3c69eceae636d203-Paper.pdf

ItstangentkernelK(x,z)(w)isdefinedasfollows: K(x,z)(w):= wf(w;x)T wf(w;z), forfixedinputsx,z Rd. (1) The key finding of [10] was the fact that for some wide neural networks the kernelK(x,z)(w) is a constant function of the weightw during training. While in the literature, including [10], this phenomenon isdescribed interms ofthe(linear) training dynamics,itisimportant tonote thatthe tangent kernel is associated to the model itself.