Exploration in environments with sparse rewards is difficult for artificial agents. Curiosity driven learning -- using feed-forward prediction errors as intrinsic rewards -- has achieved some success in these scenarios, but fails when faced with action-dependent noise sources. We present aleatoric mapping agents (AMAs), a neuroscience inspired solution modeled on the cholinergic system of the mammalian brain. AMAs aim to explicitly ascertain which dynamics of the environment are unpredictable, regardless of whether those dynamics are induced by the actions of the agent. This is achieved by generating separate forward predictions for the mean and variance of future states and reducing intrinsic rewards for those transitions with high aleatoric variance. We show AMAs are able to effectively circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents. The code for all experiments presented in this paper is open-sourced.

Jean-Pierre Changeux is France's most famous neuroscientist. Though less well known in the United States, he has directed a famous laboratory at the Pasteur Institute for more than thirty years, taught as a professor at the Collège de France, and written a number of works exploring "the neurobiology of meaning." Aside from his own books, Changeux has published two wide-ranging dialogues about mind and matter, one with the mathematician Alain Connes and the other with the late French philosopher Paul Ricoeur. Changeux came of age at a fortunate time. Born in 1936, he began his studies when the advent both of the DNA age and of high-resolution images of the brain heralded a series of impressive breakthroughs. Changeux took part in one such advance in 1965 when, together with Jacques Monod and Jeffries Wyman, he established an important model of protein interactions in bacteria, which, when applied to the brain, became crucial for understanding the behavior of neurons. Since that time, Changeux has written a number of books exploring the functions of the brain. The brain is of course tremendously complex: a bundle of some hundred billion neurons, or nerve cells, each sharing as many as ten thousand connections with other neurons.

Rapid eye movement sleep (REM sleep, REMS) is a unique phase of mammalian sleep characterized by random movement of the eyes, low muscle tone throughout the body, and the propensity of the sleeper to dream vividly. This phase is also known as paradoxical sleep (PS) and sometimes desynchronized sleep because of physiological similarities to waking states, including rapid, low-voltage desynchronized brain waves. Electrical and chemical activity regulating this phase seems to originate in the brain stem and is characterized most notably by an abundance of the neurotransmitter acetylcholine, combined with a nearly complete absence of monoamine neurotransmitters histamine, serotonin, and norepinepherine.[1] The cortical and thalamic neurons of the waking or paradoxically sleeping brain are more depolarized--i.e., can "fire" more readily--than in the deeply sleeping brain.[2] The right and left hemispheres of the brain are more coherent in REM sleep, especially during lucid dreams.[3] REM sleep is punctuated and immediately preceded by PGO (ponto-geniculo-occipital) waves, bursts of electrical activity originating in the brain stem.[4] These waves occur in clusters about every 6 seconds for 1–2 minutes during the transition from deep to paradoxical sleep.[5] They exhibit their highest amplitude upon moving into the visual cortex and are a cause of the "rapid eye movements" in paradoxical sleep.[6][7] Brain energy use in REM sleep, as measured by oxygen and glucose metabolism, equals or exceeds energy use in waking. The rate in non-REM sleep is 11–40% lower.[8]

The Charles Bonnet Syndrome (CBS) is characterized by complex vivid visual hallucinations in people with, primarily, eye diseases and no other neurological pathology. We present a Deep Boltzmann Machine model of CBS, exploring two core hypotheses: First, that the visual cortex learns a generative or predictive model of sensory input, thus explaining its capability to generate internal imagery. And second, that homeostatic mechanisms stabilize neuronal activity levels, leading to hallucinations being formed when input is lacking. We reproduce a variety of qualitative findings in CBS. We also introduce a modification to the DBM that allows us to model a possible role of acetylcholine in CBS as mediating the balance of feed-forward and feed-back processing. Our model might provide new insights into CBS and also demonstrates that generative frameworks are promising as hypothetical models of cortical learning and perception.

Combining neuropharmacological experiments with computational modeling, we have shown that cholinergic modulation may enhance associative memory function in piriform (olfactory) cortex. We have shown that the acetylcholine analogue carbachol selectively suppresses synaptic transmission between cells within piriform cortex, while leaving input connections unaffected. When tested in a computational model of piriform cortex, this selective suppression, applied during learning, enhances associative memory performance.

Combining neuropharmacological experiments with computational modeling, we have shown that cholinergic modulation may enhance associative memory function in piriform (olfactory) cortex. We have shown that the acetylcholine analogue carbachol selectively suppresses synaptic transmission between cells within piriform cortex, while leaving input connections unaffected. When tested in a computational model of piriform cortex, this selective suppression, applied during learning, enhances associative memory performance.

James M. Bower Computation and Neural Systems Caltech 216-76 Pasadena, CA 91125 Combining neuropharmacological experiments with computational modeling, wehave shown that cholinergic modulation may enhance associative memory function in piriform (olfactory) cortex. We have shown that the acetylcholine analogue carbachol selectively suppresses synaptic transmission betweencells within piriform cortex, while leaving input connections unaffected. When tested in a computational model of piriform cortex, this selective suppression, applied during learning, enhances associative memory performance.