First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. This paper introduces a framework for learning from options in reinforcement learning. An option is a policy which has some probability of terminating at a certain state. This paper introduces the notion of an "option policy", which is like a high-level policy that allows for multi-step transition between states. They show how to make the option model universal with respect to rewards, and provide an TD-style algorithm for learning with such models.

Implications for performance bounds for RL algorithms are sketched out. Empirically the new measure is demonstrated to be tighter than previously known indicators of MDP hardness.

We argue that a good solution should be able to explicitly parameterize a policy (i.e.

Hippocampal place fields have been shown to reflect behaviorally relevant aspects of space. For instance, place fields tend to be skewed along commonly traveled directions, they cluster around rewarded locations, and they are constrained by the geometric structure of the environment. We hypothesize a set of design principles for the hippocampal cognitive map that explain how place fields represent space in a way that facilitates navigation and reinforcement learning. In particular, we suggest that place fields encode not just information about the current location, but also predictions about future locations under the current transition distribution. Under this model, a variety of place field phenomena arise naturally from the structure of rewards, barriers, and directional biases as reflected in the transition policy. Furthermore, we demonstrate that this representation of space can support efficient reinforcement learning. We also propose that grid cells compute the eigendecomposition of place fields in part because is useful for segmenting an enclosure along natural boundaries. When applied recursively, this segmentation can be used to discover a hierarchical decomposition of space. Thus, grid cells might be involved in computing subgoals for hierarchical reinforcement learning.

This problem setting is motivated by the successful deep Q-networks (DQN) framework that falls in this regime.

Neural Information Processing Systems http://nips.cc/