Break and Make: Interactive Structural Understanding Using LEGO Bricks - Technology Org

The ability to reason about parts that make up objects and the structural relationships between them is essential for the ability to build tools, solve problems and manipulate the world. Firstly, a learning agent is presented with a previously unseen LEGO model and can disassemble and inspect it to learn more about its structure. Then, the agent is presented with an empty scene and must use the information gathered to rebuild the model from scratch. Researchers present a new simulator and dataset that allow interactive learning agents to build and manipulate LEGO models. Transformer-based network architecture is designed to make progress on this task.