Various algorithms in reinforcement learning exhibit dramatic variability in their convergence rates and ultimate accuracy as a function of the problem structure. Such instance-specific behavior is not captured by existing global minimax bounds, which are worst-case in nature. We analyze the problem of estimating optimal $Q$-value functions for a discounted Markov decision process with discrete states and actions and identify an instance-dependent functional that controls the difficulty of estimation in the $\ell_\infty$-norm. Using a local minimax framework, we show that this functional arises in lower bounds on the accuracy on any estimation procedure. In the other direction, we establish the sharpness of our lower bounds, up to factors logarithmic in the state and action spaces, by analyzing a variance-reduced version of $Q$-learning. Our theory provides a precise way of distinguishing "easy" problems from "hard" ones in the context of $Q$-learning, as illustrated by an ensemble with a continuum of difficulty.

Markov decision processes and reinforcement learning algorithms provide a flexible framework for decision-making in dynamic settings, and have been studied for decades (e.g., [23, 27, 8, 9, 29]). Given the explosion in the amount of available data and computing power, recent years have witnessed dramatic success of reinforcement learning (RL) techniques in various application domains (e.g., [30, 19, 26, 22, 27]). In broad terms, algorithms for reinforcement learning are often separated into model-based versus model-free approaches. Model-based approaches based on directly learning a model for the dynamics of the system, and then computing optimal policies from the learned model. In contrast, a model-free approach directly targets learning of the optimal value function or policy. Naturally, a model-free approach is more robust to model mismatch; however, model-based approaches can often be more sample efficient. Providing a firm theoretical foundation to the tradeoffs intrinsic to different classes of methods, as characterized by their access to the underlying Markov decision process, is a major open question in RL.