The Neurothermostat is an adaptive controller that regulates indoor air temperature in a residence by switching a furnace on or off. The task is framed as an optimal control problem in which both comfort and energy costs are considered as part of the control objective. Because the consequences of control decisions are delayed in time, the N eurothermostat must anticipate heating demands with predictive models of occupancy patterns and the thermal response of the house and furnace. Occupancy pattern prediction is achieved by a hybrid neural net / lookup table. The Neurothermostat searches, at each discrete time step, for a decision sequence that minimizes the expected cost over a fixed planning horizon.

Closed-loop control relies on sensory feedback that is usually assumed to be free. But if sensing incurs a cost, it may be costeffective to take sequences of actions in open-loop mode. We describe a reinforcement learning algorithm that learns to combine open-loop and closed-loop control when sensing incurs a cost. Although we assume reliable sensors, use of open-loop control means that actions must sometimes be taken when the current state of the controlled system is uncertain. This is a special case of the hidden-state problem in reinforcement learning, and to cope, our algorithm relies on short-term memory. The main result of the paper is a rule that significantly limits exploration of possible memory states by pruning memory states for which the estimated value of information is greater than its cost. We prove that this rule allows convergence to an optimal policy.

The Neurothermostat is an adaptive controller that regulates indoor air temperature in a residence by switching a furnace on or off. The task is framed as an optimal control problem in which both comfort and energy costs are considered as part of the control objective. Because the consequences of control decisions are delayed in time, the N eurothermostat must anticipate heating demands with predictive models of occupancy patterns and the thermal response of the house and furnace. Occupancy pattern prediction is achieved by a hybrid neural net / lookup table. The Neurothermostat searches, at each discrete time step, for a decision sequence that minimizes the expected cost over a fixed planning horizon.

Closed-loop control relies on sensory feedback that is usually assumed tobe free . But if sensing incurs a cost, it may be costeffective totake sequences of actions in open-loop mode. We describe a reinforcement learning algorithm that learns to combine open-loop and closed-loop control when sensing incurs a cost. Although weassume reliable sensors, use of open-loop control means that actions must sometimes be taken when the current state of the controlled system is uncertain. This is a special case of the hidden-state problem in reinforcement learning, and to cope, our algorithm relies on short-term memory.

The Neurothermostat is an adaptive controller that regulates indoor airtemperature in a residence by switching a furnace on or off. The task is framed as an optimal control problem in which both comfort and energy costs are considered as part of the control objective.Because the consequences of control decisions are delayed in time, the Neurothermostat must anticipate heating demands withpredictive models of occupancy patterns and the thermal response of the house and furnace.

Busey Bank (Champaign, Ill.) is using intelligent-agent technology to launch its Lloyds Bowmaker Motor Finance (Petersfield, U.K.) has implemented a The Philadelphia Stock Exchange care products, has developed a rulebased neural network-based system for credit (Philadelphia, Pa.) has adopted an multinational order-entry and scoring new loan applications. The company is system helps Lloyds determine whether increase the reliability and scalability using the system to process orders to accept a loan and gives the reasons of network-supported options-trading from its network of more than for its choice. The system uses an electronic facilities. The software will permit installed a rule-based expert system to camera to image the front face of letters, team members in different geographic manage the complexity of producing identify the destination address, locations to explore similar multisensory more than 20,000 new designs and and determine its delivery-point bar environments both independently 2.4 billion greeting cards annually. The company has completely reengineered its operation, converting an Telecommunications providers MCI Healthcare software developer HBO & antiquated job-shop operation into a (Washington, D.C.) and BT (London, Company (Atlanta, Ga.) is developing state-of-the-art cellular one.

In the Poisson neuron model, the output is a rate-modulated Poisson process(Snyder and Miller, 1991); the time varying rate parameter ret)is an instantaneous function G[.] of the stimulus, ret) G[s(t)]. In a Poisson neuron, then, ret) gives the instantaneous firingrate-the instantaneous probability of firing at any instant t-and the output is a stochastic function of the input. In part because of its great simplicity, this model is widely used (usually withthe addition of a refractory period), especially in in vivo single unit electrophysiological studies, where set) is usually taken to be the value of some sensory stimulus. In the integrate-and-fire neuron model, by contrast, the output is a filtered and thresholded function of the input: the input is passed through a low-pass filter (determined by the membrane time constant T) and integrated until themembrane potential vet) reaches threshold 8, at which point vet) is reset to its initial value. By contrast with the Poisson model, in the integrate-and-fire model the ouput is a deterministic function of the input. Although the integrate-and-fire model is a caricature of real neural dynamics, it captures many of the qualitative features, andis often used as a starting point for conceptualizing the biophysical behavior of single neurons.

There is currently considerable interest in developing general nonlinear densitymodels based on latent, or hidden, variables. Such models have the ability to discover the presence of a relatively small number of underlying'causes' which, acting in combination, give rise to the apparent complexity of the observed data set. Unfortunately, totrain such models generally requires large computational effort. In this paper we introduce a novel latent variable algorithm which retains the general nonlinear capabilities of previous models but which uses a training procedure based on the EM algorithm. We demonstrate the performance of the model on a toy problem and on data from flow diagnostics for a multiphase oil pipeline.

In the Poisson neuron model, the output is a rate-modulated Poisson process (Snyder and Miller, 1991); the time varying rate parameter ret) is an instantaneous function G[.] of the stimulus, ret) G[s(t)]. In a Poisson neuron, then, ret) gives the instantaneous firing rate-the instantaneous probability of firing at any instant t-and the output is a stochastic function of the input. In part because of its great simplicity, this model is widely used (usually with the addition of a refractory period), especially in in vivo single unit electrophysiological studies, where set) is usually taken to be the value of some sensory stimulus. In the integrate-and-fire neuron model, by contrast, the output is a filtered and thresholded function of the input: the input is passed through a low-pass filter (determined by the membrane time constant T) and integrated until the membrane potential vet) reaches threshold 8, at which point vet) is reset to its initial value. By contrast with the Poisson model, in the integrate-and-fire model the ouput is a deterministic function of the input. Although the integrate-and-fire model is a caricature of real neural dynamics, it captures many of the qualitative features, and is often used as a starting point for conceptualizing the biophysical behavior of single neurons.

There is currently considerable interest in developing general nonlinear density models based on latent, or hidden, variables. Such models have the ability to discover the presence of a relatively small number of underlying'causes' which, acting in combination, give rise to the apparent complexity of the observed data set. Unfortunately, to train such models generally requires large computational effort. In this paper we introduce a novel latent variable algorithm which retains the general nonlinear capabilities of previous models but which uses a training procedure based on the EM algorithm. We demonstrate the performance of the model on a toy problem and on data from flow diagnostics for a multiphase oil pipeline.