A directed generative model for binary data using a small number of hidden continuous units is investigated. The relationships between the correlations of the underly(cid:173) ing continuous Gaussian variables and the binary output variables are utilized to learn the appropriate weights of the network. The advantages of this approach are illustrated on a translationally in(cid:173) variant binary distribution and on handwritten digit images.

Many researchers have explored methods for hierarchical reinforce(cid:173) ment learning (RL) with temporal abstractions, in which abstract actions are defined that can perform many primitive actions before terminating. However, little is known about learning with state ab(cid:173) stractions, in which aspects of the state space are ignored. In this paper, we define five conditions under which state abstraction can be combined with the MAXQ value function decomposition. We prove that the MAXQ-Q learning algorithm converges under these conditions and show experimentally that state abstraction is important for the successful application of MAXQ-Q learning.

This paper examines the application of reinforcement learning to a wire(cid:173) less communication problem. The problem requires that channel util(cid:173) ity be maximized while simultaneously minimizing battery usage. We present a solution to this multi-criteria problem that is able to signifi(cid:173) cantly reduce power consumption. The solution uses a variable discount factor to capture the effects of battery usage.

Function approximation is essential to reinforcement learning, but the standard approach of approximating a value function and deter(cid:173) mining a policy from it has so far proven theoretically intractable. In this paper we explore an alternative approach in which the policy is explicitly represented by its own function approximator, indepen(cid:173) dent of the value function, and is updated according to the gradient of expected reward with respect to the policy parameters. Williams's REINFORCE method and actor-critic methods are examples of this approach. Our main new result is to show that the gradient can be written in a form suitable for estimation from experience aided by an approximate action-value or advantage function. Using this result, we prove for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy.

Recently, a number of authors have proposed treating dialogue systems as Markov decision processes (MDPs). However, the practical application ofMDP algorithms to dialogue systems faces a number of severe technical challenges. We have built a general software tool (RLDS, for Reinforcement Learning for Dialogue Systems) based on the MDP framework, and have applied it to dialogue corpora gathered from two dialogue systems built at AT&T Labs. Our experiments demonstrate that RLDS holds promise as a tool for "browsing" and understanding correlations in complex, temporally dependent dialogue corpora.

This paper proposes a new reinforcement learning (RL) paradigm that explicitly takes into account input disturbance as well as mod(cid:173) eling errors. The use of environmental models in RL is quite pop(cid:173) ular for both off-line learning by simulations and for on-line ac(cid:173) tion planning. However, the difference between the model and the real environment can lead to unpredictable, often unwanted results. Based on the theory of H oocontrol, we consider a differential game in which a'disturbing' agent (disturber) tries to make the worst possible disturbance while a'control' agent (actor) tries to make the best control input. The problem is formulated as finding a min(cid:173) max solution of a value function that takes into account the norm of the output deviation and the norm of the disturbance.

For many problems which would be natural for reinforcement learning, the reward signal is not a single scalar value but has multiple scalar com(cid:173) ponents. Examples of such problems include agents with multiple goals and agents with multiple users. Creating a single reward value by com(cid:173) bining the multiple components can throwaway vital information and can lead to incorrect solutions. We describe the multiple reward source problem and discuss the problems with applying traditional reinforce(cid:173) ment learning. We then present an new algorithm for finding a solution and results on simulated environments.

Many approaches to reinforcement learning combine neural net(cid:173) works or other parametric function approximators with a form of temporal-difference learning to estimate the value function of a Markov Decision Process. A significant disadvantage of those pro(cid:173) cedures is that the resulting learning algorithms are frequently un(cid:173) stable. In this work, we present a new, kernel-based approach to reinforcement learning which overcomes this difficulty and provably converges to a unique solution. By contrast to existing algorithms, our method can also be shown to be consistent in the sense that its costs converge to the optimal costs asymptotically. Our focus is on learning in an average-cost framework and on a practical ap(cid:173) plication to the optimal portfolio choice problem.

Visual in(cid:173) put, provided by a video camera on a miniature robot, is preprocessed by a set of Gabor filters on 31 nodes of a log-polar retinotopic graph. Place cells serve as basis func(cid:173) tions for reinforcement learning. Experimental results for goal-oriented navigation of a mobile robot are presented.

This paper presents predictive gain scheduling, a technique for simplify(cid:173) ing reinforcement learning problems by decomposition. Link admission control of self-similar call traffic is used to demonstrate the technique. The control problem is decomposed into on-line prediction of near-fu(cid:173) ture call arrival rates, and precomputation of policies for Poisson call ar(cid:173) rival processes. At decision time, the predictions are used to select among the policies. Simulations show that this technique results in sig(cid:173) nificantly faster learning without any performance loss, compared to a reinforcement learning controller that does not decompose the problem.