Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling

This is reflected in et al., 2021). Despite the success of PINNs, it is known that several recent studies on characterizing the "failure PINNs sometimes fail to converge to the correct solution modes" of PINNs, although a thorough understanding in problems involving complicated PDEs, as reflected in of the connection between PINN failure several recent studies on characterizing the "failure modes" modes and sampling strategies is missing. In of PINNs (Wang et al., 2021; 2022c; Krishnapriyan et al., this paper, we provide a novel perspective of failure 2021). Many of these failure modes are related to the susceptibility modes of PINNs by hypothesizing that training of PINNs in getting stuck at trivial solutions acting PINNs relies on successful "propagation" of as poor local minima, due to the unique optimization challenges solution from initial and/or boundary condition of PINNs. In particular, training PINNs is different points to interior points. We show that PINNs from conventional deep learning problems as we only have with poor sampling strategies can get stuck at access to the correct solution on the initial and/or boundary trivial solutions if there are propagation failures, points, while for all interior points, we can only compute characterized by highly imbalanced PDE residual PDE residuals. Also, minimizing PDE residuals does not fields. To mitigate propagation failures, we propose guarantee convergence to a correct solution since there are a novel Retain-Resample-Release sampling many trivial solutions of commonly observed PDEs that (R3) algorithm that can incrementally accumulate show 0 residuals. While previous studies have mainly focused collocation points in regions of high PDE on modifying network architectures or balancing loss residuals with little to no computational overhead.