Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks

The curvature is a key geometric property of the loss landscape, which is characterized by the Hessian matrix or approximations such as the Gauss-Newton (GN) matrix, and strongly influences the convergence of gradient-based optimization methods. In the realm of deep learning, where models often have millions of parameters, understanding the geometry of the optimization landscape is essential to understanding the effectiveness of training algorithms.