Bandit Overfitting in Offline Policy Learning

We study the offline policy learning problem in a contextual bandit framework. Specifically, we focus on the issue of overfitting which is especially important in a modern context where we often use overparameterized models that can interpolate the data. Our first contribution is to introduce a regret decomposition into approximation, estimation, and bandit errors that emphasizes the distinction between the policy learning and supervised learning problems. The bandit error measures the error from overfitting to the single action observed at each context, which we call "bandit overfitting". Our second contribution is to show both in theory and experiments how bandit overfitting is different for policy-based versus value-based algorithms when we use overparameterized models. We find that bandit overfitting can become a severe problem for policy-based algorithms, but value-based algorithms effectively reduce the policy learning problem to regression and thus avoid the worst problems of bandit overfitting.