First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. The paper is concerned with Monte Carlo sampling based on the discretisation of SDEs. This is a particularly topical subject since there has been some interest lately in such techniques due to the fact that they allow for the use of stochastic gradients which are particularly appealing in some big data settings since they allow one to run algorithms with only partial evaluation of the likelihood/energy function. The paper is particularly well written and pedagogical. In additional it clarifies earlier contributions and provides a rigorous overview of the main results useful in this emerging area.

Neural Information Processing Systems http://nips.cc/

To handle vast amounts of data, it is natural and popular to compress vectors and matrices.

Neural Information Processing Systems http://nips.cc/

The author ordering is alphabetical.

Generalization in deep learning is closely tied to the pursuit of flat minima in the loss landscape, yet classical Stochastic Gradient Langevin Dynamics (SGLD) offers no mechanism to bias its dynamics toward such low-curvature solutions. This work introduces Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD), designed to efficiently and provably seek flat minima in high-dimensional nonconvex optimization problems. At each iteration, fSGLD uses the stochastic gradient evaluated at parameters perturbed by isotropic Gaussian noise, commonly referred to as Random Weight Perturbation (RWP), thereby optimizing a randomized-smoothing objective that implicitly captures curvature information. Leveraging these properties, we prove that the invariant measure of fSGLD stays close to a stationary measure concentrated on the global minimizers of a loss function regularized by the Hessian trace whenever the inverse temperature and the scale of random weight perturbation are properly coupled. This result provides a rigorous theoretical explanation for the benefits of random weight perturbation. In particular, we establish non-asymptotic convergence guarantees in Wasserstein distance with the best known rate and derive an excess-risk bound for the Hessian-trace regularized objective. Extensive experiments on noisy-label and large-scale vision tasks, in both training-from-scratch and fine-tuning settings, demonstrate that fSGLD achieves superior or comparable generalization and robustness to baseline algorithms while maintaining the computational cost of SGD, about half that of SAM. Hessian-spectrum analysis further confirms that fSGLD converges to significantly flatter minima.

We base the justification on the following two propositions. Work developed during an internship at Borealis AI. Poisson processes for training and test. For the mixture of OU processes (MOU), we sample 5000 sequences from each of two different OU processes and mix them to obtain 10000 sequences. As mentioned in Section 5.2 of the paper, we compare our models against the baselines on three datasets: Mujoco-Hopper, Beijing Air-Quality dataset (BAQD), and PTB Diagnostic The sequence length of the Mujoco-Hopper dataset is 200 and the sequence length of BAQD is 168.