Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent Mingze Wang Massachusetts Institute of Technology, Peking University NTT Research

Symmetries are prevalent in deep learning and can significantly influence the learning dynamics of neural networks. In this paper, we examine how exponential symmetries - a broad subclass of continuous symmetries present in the model architecture or loss function - interplay with stochastic gradient descent (SGD). We first prove that gradient noise creates a systematic motion (a "Noether flow") of the parameters θ along the degenerate direction to a unique initializationindependent fixed point θ