Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our selfmonitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set). Recently, the Vision-and-Language (VLN) navigation task (Anderson et al., 2018b), which requires the agent to follow natural language instructions to navigate through a photo-realistic unknown environment, has received significant attention (Wang et al., 2018b; Fried et al., 2018). In the VLN task, an agent is placed in an unknown realistic environment and is required to follow natural language instructions to navigate from its starting location to a target location. Instead, the agent needs to be aware of its navigation status through the association between the sequence of observed visual inputs to instructions. Consider an example as shown in Figure 1, given the instruction "Exit the bedroom and go towards the table. Go to the stairs on the left of the couch. Wait on the third step.", the agent first needs to locate which instruction is needed for the next movement, which in turn requires the agent to be aware of (i.e., to explicitly represent or have an attentional focus on) which instructions were completed or ongoing in the previous steps.