A spring-block theory of feature learning in deep neural networks

Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, 306 N Wright St, Urbana, IL 61801, USA (Dated: July 30, 2024) A central question in deep learning is how deep neural networks (DNNs) learn features. This collective effect of non-linearity, noise, learning rate, width, depth, and numerous other parameters, has eluded first-principles theories which are built from microscopic neuronal dynamics. Here we present a noise-non-linearity phase diagram that highlights where shallow or deep layers learn features more effectively. We then propose a macroscopic mechanical theory of feature learning that accurately reproduces this phase diagram, offering a clear intuition for why and how some DNNs are "lazy" and some are "active", and relating the distribution of feature learning over layers with test accuracy. Deep neural networks (DNNs) progressively compute propose a macroscopic theory of feature learning in deep, features from which the final layer generates predictions.