Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators

Problem Setup We study the problem of continual real-world reinforcement learning through the lenses of a large scale experiment, where we deployed a fleet of 23 RL-enabled robots over two years in Google office buildings to sort waste and recycling. In our experiment, a robot roamed around an office building searching for "waste stations" (bins for recyclables, compost, and trash). The robot was tasked with approaching each waste station to sort it, moving items between the bins so that all recyclables (cans, bottles, etc.) were placed in the recyclable bin, all the compostable items (cardboard containers, paper cups, etc.) were placed in the compost bin, and everything else was placed in the landfill trash bin. The task of sorting waste is much harder than it sounds: not only does the robot need to correctly pick up the vast variety of objects that people deposit into waste bins, but it also needs to identify the appropriate bin for each object and sort them as quickly and efficiently as possible. The experiment setup enabled robots to learn on the job and improve through real-world experience, additional autonomous data collection in "robot classrooms," and simulation.