Figure 8: Three size variations of the multi-robot warehouse environment

A.1 Multi-Robot Warehouse The multi-robot warehouse environment (Figure 8) simulates a warehouse with robots moving and delivering requested goods. In real-world applications [38], robots pick-up shelves and deliver them to a workstation. Humans assess the content of a shelf, and then robots can return them to empty shelf locations. In this simulation of the environment, agents control robots and the action space for each agent is A = {Turn Left, Turn Right, Forward, Load/Unload Shelf} Agents can move beneath shelves when they do not carry anything, but when carrying a shelf, agents must use the corridors visible in Figure 8. The observation of an agent consists of a 3 3 square centred on the agent.