Thinking agents for zero-shot generalization to qualitatively novel tasks

Thinking agents for zero-shot generalization to qualitatively novel tasks The Obelisk Team Astera Institute Emeryville, USA Abstract Intelligent organisms can solve truly novel problems which they have never encountered before, either in their lifetime or their evolution. An important component of this capacity is the ability to "think", that is, to mentally manipulate objects, concepts and behaviors in order to plan and evaluate possible solutions to novel problems, even without environment interaction. To generate problems that are truly qualitatively novel, while still solvable zero-shot (by mental simulation), we use the combinatorial nature of environments: we train the agent while withholding a specific combination of the environment's elements. The novel test task, based on this combination, is thus guaranteed to be truly novel, while still mentally simulable since the agent has been exposed to each individual element (and their pairwise interactions) during training. We propose a method to train agents endowed with world models to make use their mental simulation abilities, by selecting tasks based on the difference between the agent's pre-thinking and post-thinking performance. When tested on the novel, withheld problem, the resulting agent successfully simulated alternative scenarios and used the resulting information to guide its behavior in the actual environment, solving the novel task in a single real-environment trial (zero-shot). 1 Introduction An important aspect of intelligence is the ability to handle novel problems. While simpler organisms are restricted to problems similar to these they have been exposed to during training, and fare badly when faced Correspondance: Thomas Miconi, thomas.miconi@gmail.comwith An major component of this capacity is the ability to think before acting. By'thinking' 1, that is, by internally manipulating concepts and behaviors and evaluating likely outcomes, agents can tackle novel problems never encountered before, by recombining existing knowledge into new solutions. This ability is perhaps the hallmark of what we think of as truly "intelligent" behavior: it is highly prevalent in humans, but is is debated whether it even exists in non-human animals [Suddendorf and Busby, 2003], including mammals such as rodents [Gillespie et al., 2021] or even great apes [Suddendorf et al., 2009, Os-vath, 2010]. Much work in machine learning has focused on training agents with increasingly complex innate behaviors.