Bridging Scene Understanding and Task Execution with Flexible Simulation Environments