[D] Relation between learning rate, batch size and gradient noise in NN? • r/MachineLearning

There's a performance tradeoff inherent in batch size selection--a larger batch size is often more efficient computationally (up to a point) but while that might increase the number of samples/s you process, it also can mean that the number of SGD iterations you take /s decreases. It currently seems that with deep networks it's preferable to take many small steps than to take fewer larger ones (again, this is a design tradeoff that requires experimentation to nail down optimally for any given dataset/net). There's an intense amount of research going on examining this phenomenon in theoretical and empirical detail, and evidence currently seems to be pointing towards the higher variance gradients of minibatch gradient descent actually being tied to generalization.