Embodied Question Answering in Photorealistic Environments with Point Cloud Perception

To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task - Embodied Question Answering [1] in photo-realistic environments (Matterport 3D). We thoroughly study navigation policies that utilize 3D point clouds, RGB images, or their combination. Our analysis of these models reveals several key findings. We find that two seemingly naive navigation baselines, forward-only and random, are strong navigators and challenging to outperform, due to the specific choice of the evaluation setting presented by [1]. We find a novel lossweighting Figure 1: We extend EmbodiedQA [1] to photorealstic environments, scheme we call Inflection Weighting to be important our agent is spawned in a perceptually and semantically when training recurrent models for navigation with behavior novel environment and tasked with answering a cloning and are able to out perform the baselines question about that environment. We examine the agent's with this technique. We find that point clouds provide a ability to navigate the environment and answer the question richer signal than RGB images for learning obstacle avoidance, by perceiving its environment through point clouds, RGB motivating the use (and continued study) of 3D deep images, or a combination of the two.