Language-Conditioned Offline RL for Multi-Robot Navigation

Natural language provides a rich and intuitive interface to describe robot tasks. For instance, commands such as "navigate to the left corner" or "pick up the can" lend themselves as more powerful and flexible alternatives to specifying (x, y) coordinates or joint configurations. Using language descriptions to specify outcomes, particularly when interfacing with a team of robots, is thus a more natural choice, and one that does not require specially-trained operators. Recent work on commanding robots with natural language tend to utilize large pretrained transformers [1] known as LLMs [2, 3, 4] or Large Multimodal Models (LMMs) [5] for both language processing and control. Often, the transformer receives a task and observation, and produces either an action or a sequence of actions to complete the task [6, 7, 8, 9]. The latter case reduces to open-loop control, which cannot adapt to uncertainty, while the former is limited by the high latency of these models, typically measured in seconds or hundreds of milliseconds, precluding them from dynamic scenarios.