Accuracy and Robustness of Weight-Balancing Methods for Training PINNs

However, like any deep learning methods, PINNs inherit stochastic properties from their underlying architecture, which can lead to challenges in convergence, sensitivity to initial conditions, and variability in performance [2]. These issues pose barriers to achieving robust and efficient training, particularly for large-scale or complex systems. Deep learning research has long recognized the impact of stochasticity on training outcomes, with factors such as parameter initialization, optimizer design, and data representation playing critical roles. For instance, the seminal work of Glorot and Bengio in [3] introduced that there are better initialization strategies than others, especially for large and deep neural networks. Based on this observation, they improved initialization schemes to address issues of vanishing or exploding gradients, significantly enhancing the training of deep neural networks. Despite these advances, PINNs are different from other classical deep learning algorithms because they consider gradients information and remain therefore susceptible to instabilities and inefficiencies during training [4, 5]. Multiple attempts have been made to improve PINNs' accuracy and efficiency, including pretraining [6, 7], reformulations of the underlying mathematical problem [8, 9], novel architectures [10, 11], new learning paradigms such as meta-learning and curriculum learning [12, 13], and loss reweighting techniques to balance competing objectives [14, 15, 16]. Because of the lack of clear metrics, all these techniques are not strictly compared, limiting their practical implementations. To address these challenges, we propose a probabilistic framework for improving the convergence properties of PINNs.