Phase transitions in deep learning The optimization process in deep learning can vary significantly in terms of smoothness and convergence rate, depending on various factors such as the complexity of the model, the quality/quantity of the data or the loss landscape characteristics. However, for non-convex problems this process has often been observed to be far from smooth and steady; instead it is rather dominated by discrete, successive phases. Recent studies have shed light on several key aspects influencing these phases and the overall optimization dynamics [1-10]. Figure 1: Phase transition in PINNs: The test error between the prediction and the exact solution converges faster after total diffusion (dashed lines), which occurs with an abrupt phase transition defined by homogeneous residuals. Although the convergence starts during the onset of the diffusion phase, the optimal training performance is met when the gradients of different batches become equivalent, indicating a general agreement on the direction of the optimizer steps (total diffusion). The importance of gradient noise in escaping local optima of non-convex optimization has been explored, demonstrating its role in guaranteeing polynomial time convergence to a global optimum [1]. The authors of the same work suggest the existence of a phase transition for a perturbed gradient descent GD algorithm, from escaping local optima to converging to a global solution as the artificial noise decreases. In a later work, a phenomenon called "super-convergence" has been highlighted, where models trained with a two-phase cyclical learning rate may lead to improved regularization balance and generalization [2]. Furthermore, recent investigations have discovered a two-phase learning regime for full-batch gradient descent (GD), characterized by distinct behaviors [3].