Appendix: LanguageandVisualEntityRelationship GraphforAgentNavigation

Directional features As in previous work [3, 6, 10], we apply a 128-dimensional directional encoding byreplicating(cosθi,sinθi,cosφi,sinφi)by32times torepresent theorientation ofeach single-viewiwith respect to the agent's current orientation, whereθi andφi are the angles of the heading and elevation to that single-view. Replicating the encoding by 32 times does not enrich its information but makes its gradient 32 times larger during back-propagation. We suspect that this benefits the agent to learn about the action-related terms (e.g.