Modeling Dynamic Environments with Scene Graph Memory

We investigate a novel instance of this problem: temporal link Embodied AI agents that search for objects in prediction with partial observability, i.e. when the past large environments such as households often need observations of the graph contain only parts of it. This to make efficient decisions by predicting object locations setting maps naturally to a common problem in embodied based on partial information. We pose this AI: using past sensor observations to predict the state of a as a new type of link prediction problem: link dynamic environment represented by a graph. Graphs are prediction on partially observable dynamic used frequently as the state representation of large scenes graphs. Our graph is a representation of a scene in the form of scene graphs (Johnson et al., 2015; Armeni in which rooms and objects are nodes, and their et al., 2019; Ravichandran et al., 2022a; Hughes et al., 2022), relationships are encoded in the edges; only parts a relational object-centric representation where nodes are of the changing graph are known to the agent at objects or rooms, and edges encode relationships such as each timestep. This partial observability poses a inside or onTop. Link prediction could be applied to challenge to existing link prediction approaches, partially observed, dynamic scene graphs to infer relationships which we address. We propose a novel state representation between pairs of objects enabling various downstream - Scene Graph Memory (SGM) - with decision-making tasks for which scene graphs have been captures the agent's accumulated set of observations, shown to be useful such as navigation (Amiri et al., 2022; as well as a neural net architecture called a Santos & Romero, 2022), manipulation (Agia et al., 2022; Node Edge Predictor (NEP) that extracts information Zhu et al., 2021) and object search (Ravichandran et al., from the SGM to search efficiently. We evaluate 2022a; Xu et al., 2022).