Robust Q-learning

A dynamic treatment strategy is a sequence of decision rules that maps individual characteristics to a treatment option at each decision point (i.e., a specific point in time in which a treatment is to be considered or altered). An optimal dynamic treatment strategy seeks to make these decisions to maximize a particular expected health outcome (Lavori & Dawson, 2000; Murphy, 2005; Nahum-Shani et al., 2012a; Lei et al., 2012; Davidian et al., 2016). This is similar to clinical decision making whereby care providers tailor the type/dose of treatment over the course of clinical care based on ongoing information regarding patient progress in treatment. The main goal of precision medicine (i.e., developing an effective dynamic treatment strategy) is to use patient characteristics to inform a personalized treatment plan as a sequence of decision rules that leads to the best possible health outcome for each patient (Nahum-Shani et al., 2012a; Chakraborty & Moodie, 2013; Moodie & Kosorok, 2015; Butler et al., 2018). Q-learning is a reinforcement learning algorithm that is widely used to estimate an optimal dynamic treatment strategy using data from multistage randomized clinical trials or observational studies (Watkins & Dayan, 1992; Nahum-Shani et al., 2012b; Laber et al., 2014).