2024 NASA SUITS Report: LLM-Driven Immersive Augmented Reality User Interface for Robotics and Space Exploration

As modern computers evolve to have better performance and larger computation power, new modes of interaction emerge along with novel designs of devices. Augmented Reality (AR), which allows users to interact with common objects via virtual interfaces, establishes a new space of designs and development and hence poses a significant challenge to machines' spatial perception of the complex real-world environment. Pose estimation of 3D objects, as one of the most commonplace and important tasks in 3D perception, is still a challenging task for modern algorithms, especially in complex real environments with alternating scene properties and irregular object geometries. Our project is at the forefront of addressing critical challenges in human-robot interaction within dynamic mobile AR environments. We focus on exploring the potential ways of interacting with robots in space, especially in a non-intrusive manner. This interaction is made possible by integrating three key components: a non-intrusive head-mounted device serving as a user interface, voice control to enable astronauts to manipulate the interface and interact with robots using verbal commands, and a robot tracking algorithm that accurately localizes the robot's position in 3D space. Enabled by these technologies, we proposed URSA, an LLM-driven immersive AR user interface for robotics and space exploration, and participated in the 2023 NASA Spacesuit User Interface Technologies for Students (NASA SUITS) [1]. This project aims to develop solutions for future spaceflight needs, particularly for the Artemis missions, which seek to establish a sustained human presence on the Moon and Mars.