HAPFI: History-Aware Planning based on Fused Information

-- Embodied Instruction Following (EIF) is a task of planning a long sequence of sub-goals given high-level natural language instructions, such as " Rinse a slice of lettuce and place on the white table next to the fork ". T o successfully execute these long-term horizon tasks, we argue that an agent must consider its past, i.e., historical data, when making decisions in each step. Nevertheless, recent approaches in EIF often neglects the knowledge from historical data and also do not e ff ectively utilize information across the modalities. T o this end, we propose History-A ware Planning based on Fused Information(HAPFI), eff ectively leveraging the historical data from diverse modalities that agents collect while interacting with the environment. Through experiments with diverse comparisons, we show that an agent utilizing historical multi-modal information surpasses all the compared methods that neglect the historical data in terms of action planning capability, enabling the generation of well-informed action plans for the next step. Moreover, we provided qualitative evidence highlighting the significance of leveraging historical multi-modal data, particularly in scenarios where the agent encounters intermediate failures, showcasing its robust re-planning capabilities. I. INTRODUCTION The substantial progress in artificial intelligence has heightened expectations for embodied agents capable of interacting with real-world environments and executing interactive actions. Consequently, ongoing research has been focusing on the development of embodied agents, including robots, with the capacity to emulate human abilities in effi ciently processing multifaceted, long-term information.