Recent experimental studies have focused on the specialization of different neural structures for different types of instrumental behavior. Recent theoretical work has provided normative accounts for why there should be more than one control system, and how the output of different controllers can be integrated. Two particlar controllers have been identified, one associated with a forward model and the prefrontal cortex and a second associated with computationally simpler, habitual, actor-critic methods and part of the striatum. We argue here for the normative appropriateness of an additional, but so far marginalized control system, associated with episodic memory, and involving the hippocampus and medial temporal cortices. We analyze in depth a class of simple environments to show that episodic control should be useful in a range of cases characterized by complexity and inferential noise, and most particularly at the very early stages of learning, long before habitization has set in. We interpret data on the transfer of control from the hippocampus to the striatum in the light of this hypothesis.

Recent experimental studies have focused on the specialization of different neural structures for different types of instrumental behavior.

Many neural areas, notably, the hippocampus, show structured, dynamical, population behavior such as coordinated oscillations.

Reinforcement learning models have long promised to unify computational, psychological and neural accounts of appetitively conditioned behavior. However, the bulk of data on animal conditioning comes from free-operant experiments measuring how fast animals will work for reinforcement. Existing reinforcement learning (RL) models are silent about these tasks, because they lack any notion of vigor. They thus fail to address the simple observation that hungrier animals will work harder for food, as well as stranger facts such as their sometimes greater productivity even when working for irrelevant outcomes such as water. Here, we develop an RL framework for free-operant behavior, suggesting that subjects choose how vigorously to perform selected actions by optimally balancing the costs and benefits of quick responding.

The misjudgement of tilt in images lies at the heart of entertaining visual illusionsand rigorous perceptual psychophysics.

Experimental data indicate that norepinephrine is critically involved in aspects of vigilance and attention. Previously, we considered the function ofthis neuromodulatory system on a time scale of minutes and longer, and suggested that it signals global uncertainty arising from gross changes in environmental contingencies. However, norepinephrine is also known to be activated phasically by familiar stimuli in welllearned tasks.Here, we extend our uncertainty-based treatment of norepinephrine tothis phasic mode, proposing that it is involved in the detection and reaction to state uncertainty within a task. This role of norepinephrine canbe understood through the metaphor of neural interrupts.

Reinforcement learning models have long promised to unify computational, psychologicaland neural accounts of appetitively conditioned behavior. However,the bulk of data on animal conditioning comes from free-operant experiments measuring how fast animals will work for reinforcement. Existingreinforcement learning (RL) models are silent about these tasks, because they lack any notion of vigor. They thus fail to address thesimple observation that hungrier animals will work harder for food, as well as stranger facts such as their sometimes greater productivity evenwhen working for irrelevant outcomes such as water. Here, we develop an RL framework for free-operant behavior, suggesting that subjects choose how vigorously to perform selected actions by optimally balancing the costs and benefits of quick responding.

As animals interact with their environments, they must constantly update estimates about their states. Bayesian models combine prior probabilities, adynamical model and sensory evidence to update estimates optimally. Thesemodels are consistent with the results of many diverse psychophysical studies. However, little is known about the neural representation andmanipulation of such Bayesian information, particularly in populations of spiking neurons. We consider this issue, suggesting a model based on standard neural architecture and activations. We illustrate theapproach on a simple random walk example, and apply it to a sensorimotor integration task that provides a particularly compelling example of dynamic probabilistic computation. Bayesian models have been used to explain a gamut of experimental results in tasks which require estimates to be derived from multiple sensory cues.

We explicitly treat two cases.

We study the synthesis of neural coding, selective attention and perceptual decision making. A hierarchical neural architecture is proposed, which implements Bayesian integration of noisy sensory input and top-down attentional priors, leading to sound perceptual discrimination. The model offers an explicit explanation for the experimentally observed modulation that prior information in one stimulus feature (location) can have on an independent feature (orientation). The network's intermediate levels of representation instantiate known physiological properties of visual cortical neurons. The model also illustrates a possible reconciliation of cortical and neuromodulatory representations of uncertainty.