All the models are trained for 200 epochs with stochastic gradient descent with a batch size = 128, momentum = 0.9, and cosine All the hyperparameters were selected with a small grid search. From epoch 150 to epoch 185 the training error of the chunks with size 128/256 decreases below 0.5%, while for smaller chunk sizes it remains above 5%. Random chunks with sizes larger than 128/256 can fit the training set, thus having the same representational power as the whole network on the training data. For W > 128/256 the test accuracy is decaying approximately with the same law as that of independent networks with the same width (see Figure 1). This picture suggests that for CIFAR100 the size of a clone is 128/256, slightly larger than the size of the clones in CIFAR10.

Third, the probabilistic bounds we obtain for SGD (i.e., on quantiles) provide novel insights over the previously known in-expectation

AI safety has long been an important issue in the deep learning community.

We initiate a formal study of reproducibility in optimization. We define a quantitative measure of reproducibility of optimization procedures in the face of noisy or error-prone operations such as inexact or stochastic gradient computations or inexact initialization.

We first introduce Equivari-ant Stein V ariational Gradient Descent algorithm - an equivariant sampling method based on Stein's identity for sampling from densities with symmetries.

Stochastic gradient descent (SGD) is a cornerstone of machine learning.