Review for NeurIPS paper: Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games

Strengths: This work extends the KG-A2C agent in several ways: First it uses attention to re-weight the different components of textual observations (e.g. The output of this first attention is then combined with another attention computed over multiple different sub-knowledge graphs corresponding to the connectivity of locations, objects in the current location, inventory, and anything that is connected to the current player. While none of the individual building blocks are particularly novel, the combination of all of these elements introduces a lot of flexibility to structurally decompose the different types of knowledge available in the game and allow the agent to pay attention to specific subsets of this knowledge. This flexibility pays dividends when it comes to the experimental evaluation and where this agent significantly improves on KG-A2C in nearly every game. The ablations presented validate that the full stacked architecture is indeed needed to maintain current levels of performance, and the analysis shows that the attention mechansims are working well insofar as they distribute attention correctly between locations descriptions and inventory contents as needed to generate the action.