RASP: A Drone-based Reconfigurable Actuation and Sensing Platform for Engaging Physical Environments with Foundation Models

Foundation models and large language models have shown immense human-like understanding and capabilities for generating text and digital media. However, foundation models that can freely sense, interact, and actuate the physical world like in the digital domain is far from being realized. This is due to a number of challenges including: 1) being constrained to the types of static devices and sensors deployed, 2) events often being localized to one part of a large space, and 3) requiring dense and deployments of devices Figure 1: RASP autonomous payload reconfiguration to achieve full coverage. As a critical step towards enabling to execute user specified task foundation models to successfully and freely interact with the physical environment, we propose RASP, a modular and Scaling up, there are few works that explore the use reconfigurable sensing and actuation platform that allows of LLMs to actuate our environments, particularly smart drones to autonomously swap onboard sensors and actuators homes [16, 17, 32], where events and actions may occur anywhere in only 25 seconds, allowing a single drone to quickly adapt in the space. These works generally focus on adapting to a diverse range of tasks. We demonstrate through real LLMs as a human-like interface to actuate common internetconnected smart home deployments that RASP enables FMs and LLMs smart appliances (e.g., speakers, television, air to complete diverse tasks up to 85% more successfully by conditioning, etc.). Much like how FMs enable general human allowing them to target specific areas with specific sensors language and sensory understanding and responses, and actuators on-the-fly.