Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter

In this work, we present a real-world robot that learns about its environment by autonomous exploration and experimentation. The robot incrementally discovers physical scene properties by performing targeted physical interactions with the scene, measuring their effects and feeding the results into a jointly-optimised neural-field MLP. With only a handful of these autonomous robotic experiments, a task-driven internal representation of the scene is built from scratch, without any pre-training or external guidance. We build upon the observation that the joint internal representation of shape and appearance learned by implicit scene representation models, and the smoothness and compactness priors present in these models, allow for ultra-efficient, scene-wide propagation of user-provided labels [2]. We remove the human from the loop entirely, presenting the first fully-autonomous, 3D scene understanding robot that combines active exploration and physical experimentation with a unified neural-field representation as its underlying computational model. The robot explores a scene with an RGB-D camera and simultaneously builds a map of semantic entropy, representing confidence in its current predictions. Considering this entropy, as well as kinematic feasibility and collision avoidance, the robot selects the optimal point of interaction and proceeds to take a physical measurement used for optimising the semantic head of the underlying MLP.