Policy evaluation with smooth and nonlinear function approximation has shown great potential for reinforcement learning. Compared to linear function approximation, it allows for using a richer class of approximation functions such as the neural networks. Traditional algorithms are based on two timescales stochastic approximation whose convergence rate is often slow.

To address the reviewers' comments, upon acceptance of this paper, we will (i) include numerical experiment Some common concerns are as follows. Details of this experiment will be found in final version. Reviewer 1: We thank the reviewer for providing constructive and supportive comments. They will be corrected in the final version. Details will be provided in the final version.

Neural Information Processing Systems http://nips.cc/

Here, we discuss some additional settings for the tabular experiments. The reason for this is that Sarsa(0.95), in contrast to MB-VI and MB-SU, is a multi-step Therefore, there is stochasticity in the update target even in deterministic environments due to exploration of the behavior policy. All methods used optimistic initialization. The pseudocode of the tabular, on-policy method used in Section 5.1 is shown in Algorithm 1. These estimates are updated at the end of the episode, using the data gathered during the episode.

This is a great development, but the lack of a consistent metric to evaluate such methods makes it difficult to compare various approaches.

Humans can quickly learn new tasks from just a handful of examples by preserving rich representations of experience [1].

The algorithm's computational complexity is polynomial in