The recent breakthroughs in the research on Large Language Models (LLMs) have triggered a transformation across several research domains. Notably, the integration of LLMs has greatly enhanced performance in robot Task And Motion Planning (TAMP). However, previous approaches often neglect the consideration of dynamic environments, i.e., the presence of dynamic objects such as humans. In this paper, we propose a novel approach to address this gap by incorporating human awareness into LLM-based robot task planning. To obtain an effective representation of the dynamic environment, our approach integrates humans' information into a hierarchical scene graph. To ensure the plan's executability, we leverage LLMs to ground the environmental topology and actionable knowledge into formal planning language. Most importantly, we use LLMs to predict future human activities and plan tasks for the robot considering the predictions. Our contribution facilitates the development of integrating human awareness into LLM-driven robot task planning, and paves the way for proactive robot decision-making in dynamic environments.

An obstacle toward construction robotization is the lack of methods to plan robot operations within the entire construction planning process. Despite the strength in modeling construction site conditions, 4D BIM technologies cannot perform construction robot task planning considering the contexts of given work environments. To address this limitation, this study presents a framework that integrates 4D BIM and robot task planning, presents an information flow for the integration, and performs high-level robot task planning and detailed simulation. The framework uniquely incorporates a construction robot knowledge base that derives robotrelated modeling requirements to augment a 4D BIM model. Then, the 4D BIM model is converted into a robot simulation world where a robot performs a sequence of actions retrieving construction-related information. A case study focusing on the interior wall frame installation demonstrates the potential of systematic integration in achieving context-aware robot task planning and simulation in construction environments. Simulated a mobile robot's actions to install wall frames in a residential building 1. Introduction Rapid advancements in robotics technologies are making the utilization of robots for dangerous, tedious, and repetitive tasks more and more practical [1]. Unlike traditional industrial robots with fixed behaviors, modern robots with mobile platforms, sensors, and actuators can be programmed to perform given tasks intelligently adapting to changing work environments. Many sectors, including manufacturing [2], rescue [3], agriculture [4], and healthcare [5], are adopting robots to automate existing processes to achieve greater productivity and safety. Many construction tasks are repetitive and labor-intensive by nature [7,8], and thus robotization of these tasks can potentially address many chronic problems, such as stagnant productivity growth [9], labor shortage [10], and work-related diseases/fatalities [11]. A growing number of robotic solutions are introduced by academic studies [12,13] and industrial applications (excavation and leveling [14], marking of layout [15], rebar tying [16], and bricklaying [17,18]). With this trend, construction sites are expected to become crowded with robots and human workers in the near future exposing human workers to robot-related hazards, such as collisions, crushing, trapping, mechanical part accidents, etc. [19]. In order to utilize robots safely and effectively in congested construction environments, both high-level task planning and detailed simulation of construction robots should be performed as part of the entire construction planning. Despite the abundant studies on the coordination between human work crews [20,21], none of the prior studies incorporated robot operations into construction planning process.

Large language models (LLMs) are accelerating the development of language-guided robot planners. Meanwhile, symbolic planners offer the advantage of interpretability. This paper proposes a new task that bridges these two trends, namely, multimodal planning problem specification. The aim is to generate a problem description (PD), a machine-readable file used by the planners to find a plan. By generating PDs from language instruction and scene observation, we can drive symbolic planners in a language-guided framework. We propose a Vision-Language Interpreter (ViLaIn), a new framework that generates PDs using state-of-the-art LLM and vision-language models. ViLaIn can refine generated PDs via error message feedback from the symbolic planner. Our aim is to answer the question: How accurately can ViLaIn and the symbolic planner generate valid robot plans? To evaluate ViLaIn, we introduce a novel dataset called the problem description generation (ProDG) dataset. The framework is evaluated with four new evaluation metrics. Experimental results show that ViLaIn can generate syntactically correct problems with more than 99% accuracy and valid plans with more than 58% accuracy.