Interview with Sherry Yang: Learning interactive real-world simulators

Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans and Pieter Abbeel won an outstanding paper award at ICLR2024 for their work Learning Interactive Real-World Simulators. In the paper, they introduce a universal simulator (called UniSim) which takes image and text input to train a robot simulator. We spoke to Sherry about this work, some of the challenges, and potential applications. There are two components – there is the universal component and then there is a simulator component. Looking at the simulator component first – typically when people build a simulator, they do this based on an understanding of the real world, using physics equations. Researchers will build a simulator to study how things work, such as how cars move, for example.