Effect of Batch Size on Neural Net Training

Welcome to the first installment in our Deep Learning Experiments series, where we run experiments to evaluate commonly-held assumptions about training neural networks. Our goal is to better understand the different design choices that affect model training and evaluation. To do so, we come up with questions about each design choice and then run experiments to answer them. In this article, we seek to better understand the impact of batch size on training neural networks. Typically, this is done using gradient descent, which computes the gradient of the loss function with respect to the parameters, and takes a step in that direction.