AgentWorld: An Interactive Simulation Platform for Scene Construction and Mobile Robotic Manipulation

Recent advancements in embodied AI and robotic manipulation have highlighted the need for scalable, interactive simulation environments that support both scene construction and data collection for training autonomous agents. While existing platforms [1, 2, 3, 4] offer partial solutions, such as scene generation[5, 3, 4] or task-specific manipulation datasets [6, 7, 8], few provide a unified framework that integrates high-fidelity scene construction with flexible mobile robotic data collection system. To bridge this gap, we present AgentWorld, an interactive simulation platform designed for procedural scene construction and mobile-based teleoperation, enabling efficient data collection for imitation learning in complex household environments. AgentWorld addresses two critical challenges in embodied AI research: (1) stable and diverse scene generation, ensuring that the simulated environments are visually realistic and physically plausible, and (2) a comprehensive data collection system in simulation, which allows seamless control of mobile bases and robotic arms for data collection. AgentWorld is built upon NVIDIA's Omniverse Isaac Sim [9] and Unreal Engine [10], allowing it to inherit both strengths including the physics engine for robot parallel training and realistic rendering effects.