In reinforcement learning, off-policy evaluation (OPE) is the problem of estimating the expected return of an evaluation policy given a fixed dataset that was collected by running one or more different policies. One of the more empirically successful algorithms for OPE has been the fitted q-evaluation (FQE) algorithm that uses temporal difference updates to learn an action-value function, which is then used to estimate the expected return of the evaluation policy. Typically, the original fixed dataset is fed directly into FQE to learn the action-value function of the evaluation policy. Instead, in this paper, we seek to enhance the data-efficiency of FQE by first transforming the fixed dataset using a learned encoder, and then feeding the transformed dataset into FQE. To learn such an encoder, we introduce an OPE-tailored state-action behavioral similarity metric, and use this metric and the fixed dataset to learn an encoder that models this metric. Theoretically, we show that this metric allows us to bound the error in the resulting OPE estimate. Empirically, we show that other state-action similarity metrics lead to representations that cannot represent the action-value function of the evaluation policy, and that our state-action representation method boosts the data-efficiency of FQE and lowers OPE error relative to other OPE-based representation learning methods on challenging OPE tasks. We also empirically show that the learned representations significantly mitigate divergence of FQE under varying distribution shifts. Our code is available here: https://github.com/Badger-RL/ROPE.

We study the problem of off-policy policy evaluation (OPPE) in RL.

In order to make counterfactual evaluations possible, a standard assumption--albeit often overlooked and unstated--is to require that the behavior policy does not depend on any unobserved variables that also affect the future states/rewards (no unobserved confounding).

Neural Information Processing Systems http://nips.cc/

In this appendix we prove Proposition 1 from Section 4. Proposition 1. We next derive two lemmas that will be used in the proofs of our theorems. Hence we select the most under-sampled action if we take!1 in Algorithm 1. Lemma 2. Let s be a state that we visit m times. The proof follows from Lemma 1. The proof is by induction.

U.S. Food and Drug Administration (FDA) approved Type-1 Diabetes Mellitus Simulator (T1DMS)

In Figure 9 we provide bias and MSE analysis of different algorithms on the domains that exhibit passive non-stationarity.

In order to make counterfactual evaluations possible, a standard assumption--albeit often overlooked and unstated--is to require that the behavior policy does not depend on any unobserved variables that also affect the future states/rewards (no unobserved confounding).