In this paper we consider an alternative strategy.

Symmetries are prevalent in deep learning and can significantly influence the learning dynamics of neural networks. In this paper, we examine how exponential symmetries - a broad subclass of continuous symmetries present in the model architecture or loss function - interplay with stochastic gradient descent (SGD). We first prove that gradient noise creates a systematic motion (a "Noether flow") of the parameters θ along the degenerate direction to a unique initialization-independent fixed point θ

The following lemma is standard in the literature, see e.g.

The Rashomon set is the set of models that are approximately as good as the best model in the class. That is, it is the set of all good models.

In this paper, we contribute to both theoretical analyses.

Neural Information Processing Systems http://nips.cc/