Incremental Learning in Mirror Flows

Neural networks trained with gradient descent often learn solutions of increasing complexity: the model first captures simple structure, then progressively incorporates finer details [AJB+17, KKN+19, ZSL25]. This incremental learning phenomenon, often visible as plateaus in the training loss separated by rapid transitions, has attracted significant theoretical attention. The most detailed analyses of incremental learning have been carried out for diagonal linear networks, including precise descriptions of transition times and plateau levels [Ber23, PF23]. This level of detail is possible because the training dynamics of these networks reduce to a mirror flow [WGL+20]. Mirror flows themselves feature incremental learning when initialized near the boundary of the domain of the mirror potential. This paper gives a rigorous description of this phenomenon for a broad class of mirror flows, thereby generalizing the previous analyses of diagonal linear networks.