Hebbian Synaptic Modifications in Spiking Neurons that Learn

In this paper, we derive a new model of synaptic plasticity, b ased on recent algorithms for reinforcement learning (in which an age nt attempts to learn appropriate actions to maximize its long-term averag e reward). We show that these direct reinforcement learning algorithms a lso give locally optimal performance for the problem of reinforcement learn ing with multiple agents, without any explicit communication between a gents. By considering a network of spiking neurons as a collection of agen ts attempting to maximize the long-term average of a reward signal, we deri ve a synaptic update rule that is qualitatively similar to Hebb's post ulate. This rule requires only simple computations, such as addition and lea ky integration, and involves only quantities that are available in the vicin ity of the synapse. Furthermore, it leads to synaptic connection strengths tha t give locally optimal values of the long term average reward. The reinforcem ent learning paradigm is sufficiently broad to encompass many learning pr oblems that are solved by the brain. We illustrate, with simulations, th at the approach is effective for simple pattern classification and motor learn ing tasks. It is widely accepted that the functions performed by neural circuits are modified by adjustments to the strength of the synaptic connectio ns between neurons. 1 In the 1940s, Donald Hebb speculated that such adjustments a re associated with simultaneous (or nearly simultaneous) firing of the presyna ptic and postsynaptic neurons [14]: When an axon of cell A ... persistently takes part in firing [cell B ], some growth process or metabolic change takes place [to incr ease] A's efficacy as one of the cells firing B .