An overview of condensation phenomenon in deep learning

Authors are listed in alphabetical order of last names. April 15, 2025 Abstract In this paper, we provide an overview of a common phenomenon, condensation, observed during the nonlinear training of neural networks: During the nonlinear training of neural networks, neurons in the same layer tend to condense into groups with similar outputs. Empirical observations suggest that the number of condensed clusters of neurons in the same layer typically increases monotonically as training progresses. Neural networks with small weight initializations or Dropout optimization can facilitate this condensation process. We also examine the underlying mechanisms of condensation from the perspectives of training dynamics and the structure of the loss landscape. The condensation phenomenon offers valuable insights into the generalization abilities of neural networks and correlates to stronger reasoning abilities in transformer-based language models. 1 Introduction Deep neural networks (DNNs) have demonstrated remarkable performance across a wide range of applications. In particular, scaling laws suggest that improvements in performance for Large Language Models (LLMs) are closely tied to the size of both the model and the dataset [KMH + 20]. Understanding how these large-scale neural networks achieve such extraordinary performance is crucial for developing principles that guide the design of more efficient, robust, and computationally cost-effective machine learning models. However, the study of large neural networks presents significant challenges, such as their enormous parameters and complex network architectures. Additionally, the data--ranging from language to image data--are often too complex to analyze using traditional methods.