Goto

Collaborating Authors

 Statistical Learning







Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent Liu Ziyin Massachusetts Institute of Technology, NTT Research

Neural Information Processing Systems

Symmetries are prevalent in deep learning and can significantly influence the learning dynamics of neural networks. In this paper, we examine how exponential symmetries - a broad subclass of continuous symmetries present in the model architecture or loss function - interplay with stochastic gradient descent (SGD). We first prove that gradient noise creates a systematic motion (a "Noether flow") of the parameters ฮธ along the degenerate direction to a unique initialization-independent fixed point ฮธ



Towards a Unified Framework of Contrastive Learning for Disentangled Representations

Neural Information Processing Systems

Contrastive learning has recently emerged as a promising approach for learning data representations that discover and disentangle the explanatory factors of the data.