We first prove the direction Z T SI(Z;T) = 0, which is equivalent to prove I(Z;T) = 0 SI(Z;T) = 0. We prove the contrapositive, i.e. rather than show LHS = RHS, we show that RHS = LHS. Now assume that supwi,vj ρ(w i Z i,v j T j) > ϵ for some i,j. Then by setting those elements in w,v unrelated to Z i,T j to zero, and those related to Z i,T j exactlythesameaswi,vj,weknowthatsupw,vρ(w Z,v T) > ϵ. All neural networks are trained by Adam with its default settings and a learning rate η = 0.001. Early stopping is an useful technique for avoiding overfitting, however it needs to be carefully considered when applied to adversarial methods.

Most existing approaches for dealing with noisy labels broadly fall into two categories: noise-modeling-based methods and memorization-effects-based methods.

In the era of large language models (LLMs), fine-tuning pretrained models has become ubiquitous. Yet the theoretical underpinning remains an open question. A central question is why only a few epochs of fine-tuning are typically sufficient to achieve strong performance on many different tasks. In this work, we approach this question by developing a statistical framework, combining rigorous early stopping theory with the attention-based Neural Tangent Kernel (NTK) for LLMs, offering new theoretical insights on fine-tuning practices. Specifically, we formally extend classical NTK theory [Jacot et al., 2018] to non-random (i.e., pretrained) initializations and provide a convergence guarantee for attention-based fine-tuning. One key insight provided by the theory is that the convergence rate with respect to sample size is closely linked to the eigenvalue decay rate of the empirical kernel matrix induced by the NTK. We also demonstrate how the framework can be used to explain task vectors for multiple tasks in LLMs. Finally, experiments with modern language models on real-world datasets provide empirical evidence supporting our theoretical insights.