A Theoretical Framework of Almost Hyperparameter-free Hyperparameter Selection Methods for Offline Policy Evaluation

Offline policy evaluation (OPE) is an indispensable component of the offline reinforcement learning (RL), which is a variant of reinforcement learning with special emphasis on cost-sensitive real-life applications (Levine et al., 2020), such as autonomous vehicles, finance, healthcare and molecular discovery. Almost all the offline RL algorithms involve their own hyperparameters. For example, if we employ neural networks in policy learning, we have to at least decide the network topology (e.g., number of neurons and layers, to use the residual connection or not, to use the dense connection or the convolution), the activation functions, regularization weights and the optimizers (e.g., SGD or Adam with their own hyperparameter choices). The choice of the models such as neural network is also considered to be a hyperparameters. OPE allows us to optimize or validate the choices over these hyperparameters based only on offline datasets, i.e., without access to environment simulators.