Over the past decade or so, advancements in machine learning have enabled the development of systems that are increasingly autonomous, including self-driving vehicles, virtual assistants and mobile robots. Among other things, researchers developing autonomous systems need to identify ways to integrate components designed to tackle different and yet complementary sub-tasks. For instance, a robot that completes manual tasks in a human user's home should be able to sense objects in its environment while also retrieving information about these objects that can then be used to plan its movements and actions. This process, also known as the "perception-cognition-action" paradigm, is of crucial importance, as it ultimately allows the robot to come up with useful strategies and efficiently complete tasks. So far, most methods to implement this perception-cognition-action paradigm in robots treat these three tasks as almost entirely independent modules that act as black boxes for one another.