Pixels-to-Graph: Real-time Integration of Building Information Models and Scene Graphs for Semantic-Geometric Human-Robot Understanding

-- Autonomous robots are increasingly playing key roles as support platforms for human operators in high-risk, dangerous applications. T o accomplish challenging tasks, an efficient human-robot cooperation and understanding is required. While typically robotic planning leverages 3D geometric information, human operators are accustomed to a high-level compact representation of the environment, like top-down 2D maps representing the Building Information Model (BIM). In this work, we introduce Pixels-to-Graph (Pix2G), a novel lightweight method to generate structured scene graphs from image pixels and LiDAR maps in real-time for the autonomous exploration of unknown environments on resource-constrained robot platforms. T o satisfy onboard compute constraints, the framework is designed to perform all operation on CPU only. The method output are a de-noised 2D top-down environment map and a structure-segmented 3D pointcloud which are seamlessly connected using a multi-layer graph abstracting information from object-level up to the building-level. The proposed method is quantitatively and qualitatively evaluated during real-world experiments performed using the NASA JPL NeBula-Spot legged robot to autonomously explore and map cluttered garage and urban office like environments in real-time. I. INTRODUCTION Autonomous mobile robots are increasingly utilized for augmenting human actions in everyday operations. Given their maturing abilities to robustly carry out complex tasks in dynamic and challenging environments, they are especially being deployed in dirty and dangerous applications where the risk to human lives is high. Nevertheless, in applications like infrastructure inspection and disaster response, robotic autonomy still needs human operator support for carrying out the complex decision making process. The decision making process is typically guided by the situational awareness provided by the robot and transmitted to human operators: detailed and time-critical situational awareness provision leads to more accurate and efficient mission strategies.