Provable Generalization Bounds for Deep Neural Networks with Momentum-Adaptive Gradient Dropout

Deep neural networks (DNNs) have revolutionized machine learning, achieving unprecedented success in tasks such as image classification [12], natural language processing [20], and reinforcement learning [17]. However, their overparameterized nature often leads to overfitting, where models excel on training data but fail to generalize to unseen samples [23]. This generalization gap defined as the difference between training and test error poses a critical challenge, particularly in high-stakes applications like medical diagnostics or autonomous systems [7]. Regularization techniques, such as dropout [18] and weight decay, are widely used to mitigate overfitting, but their static nature limits adaptability to the complex, non-convex loss landscapes of DNNs. Recent advances in adaptive regularization [14, 5] show promise by dynamically adjusting parameters during training, yet these methods often lack rigorous theoretical guarantees to quantify their impact on generalization.