Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity

With the recent evolution of mobile health technologies, health scientists are increasingly interested in delivering interventions via notifications on mobile device at the moments when they can most readily help the user prevent negative health outcomes and promote the adoption and maintenance of healthy behaviors. The type and timing of the mobile health interventions should ideally adapt to the real-time collected user's context, e.g., the time of the day, the location, current activity and stress level. This gives rise to the concept of a justin-time adaptive intervention (JITAI) [28]. Operationally, JITAI includes a sequence of decision rules (e.g., treatment policy) that takes the user's current context as input and specifies whether and what type of an intervention should be provided at the moment. In practice, behavioral theory along with expert opinion and analyses of existing data is often used to design the decision rules. However, these theories are often insufficiently mature to precisely specify which particular intervention and when it should be delivered in order to ensure the interventions have the intended effects and optimize the long-term efficacy of the interventions. As a result, there is much interest in how best to use data to inform the design of JITAIs [12, 39, 3, 35, 26, 41, 33, 10, 34, 42] This paper develops a Reinforcement Learning (RL) algorithm to continuously learn, e.g., online, and optimize the treatment policy in the JITAI as the user experiences the intervention.