Reinforcement learning-based estimation for partial differential equations

We evaluate the state estimation performance of the RL-ROE for systems governed by the Burgers equation and Navier-Stokes equations. For each system, we first compute various solution trajectories corresponding to different physical parameter values, which we use to construct the ROM and train the RL-ROE following the procedure outlined in Section 2.4. The trained RL-ROE is finally deployed online and compared against a time-dependent Kalman filter constructed from the same ROM, which we refer to as KF-ROE. The KF-ROE is given by equations (3a) and (4), with the calculation of the time-varying Kalman gain detailed in Appendix C of the supplementary materials. Before proceeding to the results, we discuss our choice of baseline. The ensemble Kalman filter and 4D-Var are two estimation techniques for high-dimensional systems such as those governed by PDEs (Lorenc, 2003). Although they are commonly employed for data assimilation in numerical weather prediction, they require large computational resources since they involve repeated solutions of the high-dimensional dynamics (1). Thus, they are not applicable in the context of embedded control systems, whose limited resources call for an inexpensive model such as the ROM (2). Since the ROM that we consider has linear dynamics, extensions of the Kalman filter for nonlinear dynamics such as the extended or unscented Kalman filters (Wan & Van Der Merwe, 2000; Julier & Uhlmann, 2004) are not relevant, and the vanilla Kalman filter remains the best choice of baseline.