Bayesian Reinforcement Learning with Limited Cognitive Load

Cognitive science aims to identify the principles and mechanisms that underlie adaptive behavior. An important part of this endeavor is the development of unifying, normative theories that specify "design principles" that guide or constrain how intelligent systems respond to their environment [Marr, 1982, Anderson, 1990, Lewis et al., 2014, Griffiths et al., 2015, Gershman et al., 2015]. For example, accounts of learning, cognition, and decision-making often posit a function that an organism is optimizing--e.g., maximizing long-term reward or minimizing prediction error--and test plausible algorithms that achieve this--e.g., a particular learning rule or inference process. Historically, normative theories in cognitive science have been developed in tandem with new formal approaches in computer science and statistics. This partnership has been fruitful even given differences in scientific goals (e.g., engineering artificial intelligence versus reverse-engineering biological intelligence). Normative theories play a key role in facilitating cross-talk between different disciplines by providing a shared set of mathematical, analytical, and conceptual tools for describing computational problems and how to solve them [Ho and Griffiths, 2022]. This paper is written in the spirit of such cross-disciplinary fertilization. Here, we review recent work in computer science [Arumugam and Van Roy, 2021a, 2022] that develops a novel approach for unifying three distinct mathematical frameworks that will be familiar to many cognitive scientists (Figure 1).