Credit assignment, i.e. determining how to correctly associate delayed rewards with states or state-action pairs, is a crucial problem for reinforcement learning (RL) agents ( Sutton and Barto, 2018).

We consider the problem of efficient credit assignment in reinforcement learning.

We thank all reviewers for their valuable feedback. We believe our results make a significant contribution to the field of theoretical reinforcement learning. Therefore, analyzing a variant of Nash Q-learning may be of independent interest. Since NE always exists, CCE always exists, i.e., the set of linear constraints are always feasible. The "hat" version is the actual certified policy (which can be executed as in Algorithm 2 and 4).

Existing inverse reinforcement learning (IRL) algorithms have assumed each expert's demonstrated trajectory to be produced by only a single reward function. This paper presents a novel generalization of the IRL problem that allows each trajectory to be generated by multiple locally consistent reward functions, hence catering to more realistic and complex experts' behaviors. Solving our generalized IRL problem thus involves not only learning these reward functions but also the stochastic transitions between them at any state (including unvisited states). By representing our IRL problem with a probabilistic graphical model, an expectation-maximization (EM) algorithm can be devised to iteratively learn the different reward functions and the stochastic transitions between them in order to jointly improve the likelihood of the expert's demonstrated trajectories. As a result, the most likely partition of a trajectory into segments that are generated from different locally consistent reward functions selected by EM can be derived. Empirical evaluation on synthetic and real-world datasets shows that our IRL algorithm outperforms the state-of-the-art EM clustering with maximum likelihood IRL, which is, interestingly, a reduced variant of our approach.

We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs).

Embodied agents operating in human spaces must be able to master how their environment works: what objects can the agent use, and how can it use them? We introduce a reinforcement learning approach for exploration for interaction, whereby an embodied agent autonomously discovers the affordance landscape of a new unmapped 3D environment (such as an unfamiliar kitchen).

Generating good candidate actions is critical to the success of sample-based planners, particularly in continuous or large action spaces.