Reviews: Towards Interpretable Reinforcement Learning Using Attention Augmented Agents

The paper is well-written and clear; the architecture is described in detail through a diagram (Figure 1 on page 2), with the math in section 2 expanding on the key components of the attention mechanism. High-level details for the RL training setup, implemented baselines, and condensed results are provided in the body of the paper. Detailed learning curves for each of the compared approaches are presented in the appendix (which is appropriate, given that the task-specific learning performance is secondary to the analysis of the attention mechanism). The analysis section is thorough, and I specifically appreciated the section at the end comparing the learned attention mechanism to prior work on saliency maps. Model/Architecture Notes: While the proposed model is a straightforward extension of query-key-value attention to tasks in RL, there are two interesting architectural features: First, "queries" for their attention mechanism can be decomposed into features that act on content (which the paper refers to as the "what"), and features that act on spatial location (which the paper refers to as the "where").