Robots can be designed to clean the floor, as long as everything they might encounter in the room has a programmed response. Suddenly introduce an open bottle of juice and the robot is likely to knock it over then smear juice all over the floor, completely unaware that it is making the situation worse. Robots can be designed to answer questions from humans, as long as every question has a programmed response – ask something new and the robot will have very little to say. Robots can even be designed to dance, but try to dance with it and we are likely to get hurt as the robot cannot read and react to our unpredictable human dance moves. This is the stage robotics is at today, but recent developments in artificial intelligence (AI) may now open the door to robots that can whirl you round the dancefloor, charm you with natural conversation, and even clean up the unpredictable mess afterwards.

Facebook has introduced Droidlet, an open-source, modular, heterogeneous embodied agent architecture. The Droidlet platform can be used to build embodied agents using natural language processing, computer vision, and robotics. Now with Facebook Droidlet platform, researchers can build more intelligent real-world robots. In addition, it simplifies the integration of a wide range of state-of-the-art machine learning algorithms and robotics to facilitate rapid prototyping. A droid agent is considered to be made up of a collection of components, which are both heuristic and learned.

In recent years, there have been significant advances in building end-to-end Machine Learning (ML) systems that learn at scale. But most of these systems are: (a) isolated (perception, speech, or language only); (b) trained on static datasets. On the other hand, in the field of robotics, large-scale learning has always been difficult. Supervision is hard to gather and real world physical interactions are expensive. In this work we introduce and open-source droidlet, a modular, heterogeneous agent architecture and platform. It allows us to exploit both large-scale static datasets in perception and language and sophisticated heuristics often used in robotics; and provides tools for interactive annotation. Furthermore, it brings together perception, language and action onto one platform, providing a path towards agents that learn from the richness of real world interactions.