We then give a practical approximation using neural networks anddemonstrate itsperformance andsampleefficiencyinpractice.

Insomework,functionapproximation scheme is adopted such that essential quantities for policy improvement, e.g.

Then, we consider sufficient assumptions under which learning good policies requires polynomial number of episodes.

Note that inthis simple example, the state transition functionT and the reward functionr stillsatisfy theMarkovproperty.

O((|S ||A|+ T) H ( log ( 1/")/")). ItisalsoeasyV ( , )andQ ( , , )obtained algorithm.

There, areward function isdrawn from one of multiple possible reward models atthebeginning ofeveryepisode, buttheidentity ofthechosen rewardmodel is not revealed to the agent. Hence, the latent state space, for which the dynamics are Markovian, is not given to the agent. We study the problem of learning a near optimal policy for two reward-mixing MDPs. Unlike existing approaches that rely on strong assumptions on the dynamics, we make no assumptions and study the problem in full generality.

In online learning problems, exploiting low variance plays an important role in obtaining tight performance guarantees yet ischallenging because variances are often not known a priori. Recently, considerable progress has been made by Zhangetal.

We thank the reviewers for their thoughtful reviews; below we address their main concerns. This allows us to express the misspecification error (e.g., eqn 37 in appendix) directly in every (null 1) Note that the results from Chi et al. We consider this work as a first step in this direction. Is a good representation sufficient for sample efficient reinforcement learning?

A recurring theme in statistical learning, online learning, and beyond is that faster convergence rates are possible for problems with low noise, often quantified by the performance of the best hypothesis; such results are known as first-order or small-loss guarantees. While first-order guarantees are relatively well understood in statistical and online learning, adapting to low noise in contextual bandits (and more broadly, decision making) presents major algorithmic challenges. In a COLT 2017 open problem, Agarwal, Krishnamurthy, Langford, Luo, and Schapire asked whether first-order guarantees are even possible for contextual bandits and---if so---whether they can be attained by efficient algorithms. We give a resolution to this question by providing an optimal and efficient reduction from contextual bandits to online regression with the logarithmic (or, cross-entropy) loss. Our algorithm is simple and practical, readily accommodates rich function classes, and requires no distributional assumptions beyond realizability.