In the standard MLE setting [see, e.g., Murphy, 2012, Ch. 9] we are interested in learning the These two definitions are, however, essentially equivalent. Eq. (15) is a smooth objective that can be optimized with a (stochastic) gradient descent procedure. This section contains the proofs of the results relative to the perturb and map section (Section 3.2) and The proposition now follows from arguments made in Papandreou and Y uille [2011] Its moment generating function has the form E[exp(tX)] = Γ(1 τt). As mentioned in Johnson and Balakrishnan [p. Parts of the proof are inspired by a post on stackexchange Xi'an [2016].Theorem 1.

On the pessimistic side, the paper suggests that such results are fragile.

There has been a great empirical progress in robust training of neural networks against noisy labels.

Our derivation of the update for IL (Eq. 3) is based on an expansion of log We examine the consequences of this bias formula for our specific model. Note that the update term in Eq. (S1) is However, we will show in Appendix C that these updates may have high variance. 'reparameterization trick,' in which a change of variables allows the use of stochastic gradient descent It is worth noting that this'reparameterization' will work only for additive Gaussian noise. As already mentioned, WS can be viewed as a special case of IL. Since WS is a special case of IL, the bias properties of its individual samples are identical.

Proving algorithm-dependent generalization error bounds for gradient-type optimization methods has attracted significant attention recently in learning theory. However, most existing trajectory-based analyses require either restrictive assumptions on the learning rate (e.g., fast decreasing learning rate), or continuous injected

Recent findings demonstrate that modern neural networks trained by full-batch gradient descent typically enter a regime called Edge of Stability (EOS).

This paper investigates the asymmetric low-rank matrix completion problem, which can be formulated as an unconstrained non-convex optimization problem with a nonlinear least-squares objective function, and is solved via gradient descent methods. Previous gradient descent approaches typically incorporate regularization terms into the objective function to guarantee convergence. However, numerical experiments and theoretical analysis of the gradient flow both demonstrate that the elimination of regularization terms in gradient descent algorithms does not adversely affect convergence performance. By introducing the leave-one-out technique, we inductively prove that the vanilla gradient descent with spectral initialization achieves a linear convergence rate with high probability. Besides, we demonstrate that the balancing regularization term exhibits a small norm during iterations, which reveals the implicit regularization property of gradient descent. Empirical results show that our algorithm has a lower computational cost while maintaining comparable completion performance compared to other gradient descent algorithms.

This paper introduces Domain Generalization Sharpness-Aware Minimization Model-Agnostic Meta-Learning (DGS-MAML), a novel meta-learning algorithm designed to generalize across tasks with limited training data. DGS-MAML combines gradient matching with sharpness-aware minimization in a bi-level optimization framework to enhance model adaptability and robustness. We support our method with theoretical analysis using PAC-Bayes and convergence guarantees. Experimental results on benchmark datasets show that DGS-MAML outperforms existing approaches in terms of accuracy and generalization. The proposed method is particularly useful for scenarios requiring few-shot learning and quick adaptation, and the source code is publicly available at GitHub.