Missingness as Stability: Understanding the Structure of Missingness in Longitudinal EHR data and its Impact on Reinforcement Learning in Healthcare

There is an emerging trend in the reinforcement learning for healthcare literature. In order to prepare longitudinal, irregularly sampled, cli nical datasets for reinforcement learning algorithms, many researchers will resa mple the time series data to short, regular intervals and use last-observation- carried-forward (LOCF) imputation to fill in these gaps. Typically, they will not mai ntain any explicit information about which values were imputed. In this work, w e (1) call attention to this practice and discuss its potential implication s; (2) propose an alternative representation of the patient state that addresses som e of these issues; and (3) demonstrate in a novel but representative clinical data set that our alternative representation yields consistently better results for ach ieving optimal control, as measured by off-policy policy evaluation, compared to repr esentations that do not incorporate missingness information.