Chaotic Dynamics are Intrinsic to Neural Network Training with SGD

With the advent of deep learning over the last decade, a considerable amount of effort has gone into better understanding and enhancing Stochastic Gradient Descent so as to improve the performance and stability of artificial neural network training.