A New Study by Google and DeepMind Introduces Geometric Complexity (GC) for Neural Network Analysis and Understanding of Deep Learning Models

Understanding how regularisation affects the properties of the learned solution is a blooming research topic. This is a particularly crucial component of deep learning. Whether we include it explicitly as a penalty term in a loss function or implicitly through the choice of hyperparameters, model architecture, or initialization, regularisation can take many shapes. In practice, regularisation is routinely used to control model complexity, putting pressure on a model to identify simple solutions rather than complicated answers, even though these forms are not often intended to be analytically tractable. There is a need for a clear definition of model "complexity" for deep neural networks to comprehend regularisation in deep learning.