Interpretable Robot Control via Structured Behavior Trees and Large Language Models

With the increasing presence of intelligent robots in everyday life, the demand for reliable and straightforward Human-Robot Interaction (HRI) interfaces is rapidly rising. Traditional robot control paradigms require users to learn particular commands [1] or interact with the robots through rigid user interfaces, especially in unstructured environments [2]. However, recent works target more flexible and adaptive communication strategies, unlocking the full potential of autonomous agents in human-centered environments. Accordingly, advances in generative AI and Large Language Models (LLMs) reveal new opportunities for enabling seamless communication between humans and robots, where natural language is the primary means of communication [3]. Such models are powerful enough to comprehend given instructions and even "reason" about the demanded tasks, intentions, and environmental context [4]. When paired with robotic perception and control systems, LLMs enable users to intuitively instruct the robot to perform complex tasks such as following multiple objects [5], navigating through dynamic scenes [6], or interacting with specific items [7], all using natural dialogue. Furthermore, integrating multimodal capabilities, including vision and speech, enhances HRI by enabling more natural, context-aware communication and improving adaptability across tasks and environments [8].