In real-world healthcare problems, there are often multiple competing outcomes of interest, such as treatment efficacy and side effect severity. However, statistical methods for estimating dynamic treatment regimes (DTRs) usually assume a single outcome of interest, and the few methods that deal with composite outcomes suffer from important limitations. This includes restrictions to a single time point and two outcomes, the inability to incorporate self-reported patient preferences and limited theoretical guarantees. To this end, we propose a new method to address these limitations, which we dub Latent Utility Q-Learning (LUQ-Learning). LUQ-Learning uses a latent model approach to naturally extend Q-learning to the composite outcome setting and adopt the ideal trade-off between outcomes to each patient. Unlike previous approaches, our framework allows for an arbitrary number of time points and outcomes, incorporates stated preferences and achieves strong asymptotic performance with realistic assumptions on the data. We conduct simulation experiments based on an ongoing trial for low back pain as well as a well-known completed trial for schizophrenia. In all experiments, our method achieves highly competitive empirical performance compared to several alternative baselines.

Unfortunately, the best policy from a set of many policies such estimates are often inaccurate (Fu et al., 2021). As given only logged data, is crucial for applying an alternative, many works have explored using empirical offline RL in real-world settings. One idea that Bellman errors to perform OMS, but have found them to has been extensively explored is to select policies be poor predictors of value model accuracy (Irpan et al., based on the mean squared Bellman error 2019; Paine et al., 2020). This has led to a belief among (MSBE) of the associated Q-functions. However, many researchers that Bellman errors are not useful for previous work has struggled to obtain adequate OMS (Géron, 2019; Fujimoto et al., 2022). OMS performance with Bellman errors, leading many researchers to abandon the idea. To this end, To this end, we propose a new algorithm, Supervised Bellman we elucidate why previous work has seen pessimistic Validation (SBV), that provides a better proxy for the results with Bellman errors and identify true Bellman errors than empirical Bellman errors. SBV conditions under which OMS algorithms based achieves strong performance on diverse tasks ranging from on Bellman errors will perform well. Moreover, healthcare problems (Klasnja et al., 2015) to Atari games we develop a new estimator of the MSBE that is (Bellemare et al., 2013).