Many Episode Learning in a Modular Embodied Agent via End-to-End Interaction

In this work we give a case study of a modular embodied machine-learning (ML) powered agent that improves itself via interactions with crowd-workers. The agent consists of a set of modules, some of which are learned, and others heuristic. While the agent is not "end-to-end" in the ML sense, end-to-end interaction with humans and its environment is a vital part of the agent's learning mechanism. We describe how the design of the agent works together with the design of multiple annotation interfaces to allow crowd-workers to assign credit to module errors from these end-toend interactions, and to label data for an individual module. We further show how this whole loop (including model re-training and re-deployment) can be automated. Over multiple loops with crowdsourced humans with no knowledge of the agent architecture, we demonstrate improvement over the agent's language understanding and visual perception modules. Present day machine learning (ML) research prioritizes end-to-end learning. Not only are end-to-end models able to achieve excellent performance on static tasks, there is a growing literature on how to adapt pre-trained networks to new tasks, and large pre-trained models can have impressive zero-shot performance on unseen tasks. In the setting of embodied agents, this manifests as agents actualized as monolithic ML models, where inputs to the model are the agent's perceptual sensors, and the model's outputs directly control agent actions. There are now a number of environments designed for the training of end-to-end embodied agents Beattie et al. (2016); Savva et al. (2019); Guss et al. (2019); Petrenko et al. (2021), and there is hope (and some evidence) that the same sort of transfer and adaptability seen in language and vision models will carry over to the embodied agent setting. Nevertheless, agents implemented as fully end-to-end ML models are rare in production systems (or in real-world embodied agents, a.k.a. While this in part is a symptom of the rapid improvement and scaling in the literature and the lag in technology transfer, these systems require performance and safety guarantees that are still not easily obtainable from end-to-end ML models; and must be maintainable by human engineers.