Treating Epilepsy by Reinforcement Learning Via Manifold-Based Simulation

The ability to take intelligent actions in real-world domains is a goal of great interest in the machine learning community. Unfortunately, the real-world is filled with systems that can bepartially observed but cannot, as yet, be described by first principlemodels. Moreover, the traditional paradigm of direct interaction with the environment used in reinforcement learning (RL) is often prohibitively expensive in practice. An alternative approach that simultaneously solves both of these problems is to gain experience in simulation; the simulation in this approach is a computational model derived from observations. Advances in sensory and information technology are simplifying the acquisition and distribution of real-world datasets to computational scientists; thus, the barrier to linking intelligent control with real-world domains is becoming one of identifying high-quality state-space and transition functions directly from observations. From a dynamical systems perspective, this barrier is analogous to the problem of finding high-quality manifold embeddings and a rich literature of theory and practice exists to address it. The contribution of this work is two-fold. First, we describe an approach for learning optimal control strategies directly from observations using manifold embeddings as the intermediate state representation. Second, we demonstrate how control strategies constructed in this way can answer important scientific questions. As a concrete example, we use our approach to guide experimental decisions in neurostimulation treatments of epilepsy.