SayTap: Language to Quadrupedal Locomotion

Simple and effective interaction between human and quadrupedal robots paves the way towards creating intelligent and capable helper robots, forging a future where technology enhances our lives in ways beyond our imagination [1, 2, 3]. Key to such human-robot interaction system is enabling quadrupedal robots to respond to natural language instructions as language is one of the most important communication channels for human beings. Recent developments in Large Language Models (LLMs) have engendered a spectrum of applications that were once considered unachievable, including virtual assistance [4], code generation [5], translation [6], and logical reasoning [7], fueled by the proficiency of LLMs to ingest an enormous amount of historical data, to adapt in-context to novel tasks with few examples, and to understand and interact with user intentions through a natural language interface. The burgeoning success of LLMs has also kindled interest within the robotics researcher community, with an aim to develop interactive and capable systems for physical robots [8, 9, 10, 11, 12, 13]. Researchers have demonstrated the potential of using LLMs to perform high-level planning [8, 9], and robot code writing [11, 13]. Nevertheless, unlike text generation where LLMs directly interpret the atomic elements--tokens--it often proves challenging for LLMs to comprehend low-level robotic commands such as joint angle targets or motor torques, especially for inherently unstable legged robots necessitating high-frequency control signals. Consequently, most existing work presume the provision of high-level APIs for LLMs to dictate robot behaviour, inherently limiting the system's expressive capabilities. We address this limitation by using foot contact patterns as an interface that bridges human instructions in natural language and low-level commands.