We extend the provably convergent Full Gradient DQN algorithm for discounted reward Markov decision processes from Avrachenkov et al. (2021) to average reward problems. We experimentally compare widely used RVI Q-Learning with recently proposed Differential Q-Learning in the neural function approximation setting with Full Gradient DQN and DQN. We also extend this to learn Whittle indices for Markovian restless multi-armed bandits. We observe a better convergence rate of the proposed Full Gradient variant across different tasks.

Join the Discord Server to collaborate. This is a Deep Reinforcement Learning library focused on improving development by introducing RLOps - MLOps for reinforcement learning. This library is initially focused on reducing the time taken for training models and hyperparameter optimization (HPO) by pioneering evolutionary HPO techniques for reinforcement learning. Evolutionary HPO has been shown to drastically reduce overall training times by automatically converging on optimal hyperparameters, without requiring numerous training runs. We are constantly adding more algorithms, with a view to add hierarchical and multi-agent algorithms soon.

Dyna architectures (Sutton, 1990) use learning algorithms to approximate the con(cid:173) ventional optimal control technique known as dynamic programming (DP) (Bell(cid:173) man, 1957; Bertsekas, 1987). DP itself is not a learning method, but rather a computational method for determining optimal behavior given a complete model of the task to be solved. It is very similar to state-space search, but differs in that it is more incremental and never considers actual action sequences explicitly, only single actions at a time. This makes DP more amenable to incremental planning at execution time, and also makes it more suitable for stochastic or incompletely modeled environments, as it need not consider the extremely large number of se(cid:173) quences possible in an uncertain environment.

Barto, Sutton and Watkins [2] introduced a grid task as a didactic ex(cid:173) ample of temporal difference planning and asynchronous dynamical pre (cid:173) gramming. This paper considers the effects of changing the coding of the input stimulus, and demonstrates that the self-supervised learning of a particular form of hidden unit representation improves performance.

This work addresses three problems with reinforcement learning and adap(cid:173) tive neuro-control: 1. Non-Markovian interfaces between learner and en(cid:173) vironment. An algorithm is described which is based on system realization and on two interacting fully recurrent continually running net(cid:173) works which may learn in parallel. Problems with parallel learning are attacked by'adaptive randomness'. It is also described how interacting model/controller systems can be combined with vector-valued'adaptive critics' (previous critics have been scalar).

We present an algorithm based on reinforcement and state recurrence learning techniques to solve control scheduling problems. In particular, we have devised a simple learning scheme called "handicapped learning", in which the weights of the associative search element are reinforced, either positively or negatively, such that the system is forced to move towards the desired setpoint in the shortest possible trajectory. To improve the learning rate, a variable reinforcement scheme is employed: negative reinforcement values are varied depending on whether the failure occurs in handicapped or normal mode of operation. Furthermore, to realize a simulated annealing scheme for accelerated learning, if the system visits the same failed state successively, the negative reinforcement value is increased. In examples studied, these learning schemes have demonstrated high learning rates, and therefore may prove useful for in-situ learning.

A method is described for generating plan-like. The experiments reported here use a simulated vehicle with a primitive range sensor. Avoidance behaviour is encoded as a set of continuous functions of the perceptual input space. These functions are stored using CMACs and trained by a variant of Barto and Sutton's adaptive critic algorithm. As the vehicle explores its surroundings it adapts its responses to sensory stimuli so as to minimise the negative reinforcement arising from collisions.

This paper examines whether temporal difference methods for training connectionist networks, such as Suttons's TO(') algorithm, can be suc(cid:173) cessfully applied to complex real-world problems. A number of important practical issues are identified and discussed from a general theoretical per(cid:173) spective. These practical issues are then examined in the context of a case study in which TO(') is applied to learning the game of backgammon from the outcome of self-play. This is apparently the first application of this algorithm to a complex nontrivial task. It is found that, with zero knowledge built in, the network is able to learn from scratch to play the entire game at a fairly strong intermediate level of performance, which is clearly better than conventional commercial programs, and which in fact surpasses comparable networks trained on a massive human expert data set.

One way to speed up reinforcement learning is to enable learning to happen simultaneously at multiple resolutions in space and time. This paper shows how to create a Q-Iearning managerial hierarchy in which high level managers learn how to set tasks to their sub(cid:173) managers who, in turn, learn how to satisfy them. Sub-managers need not initially understand their managers' commands. They simply learn to maximise their reinforcement in the context of the current command. We illustrate the system using a simple maze task .. As the system learns how to get around, satisfying commands at the multiple levels, it explores more efficiently than standard, flat, Q-Iearning and builds a more comprehensive map.

In stochastic learning, weights are random variables whose time evolution is governed by a Markov process. At each time-step, n, the weights can be described by a probability density function pew, n). We summarize the theory of the time evolution of P, and give graphical examples of the time evolution that contrast the behavior of stochastic learning with true gradient descent (batch learning). Finally, we use the formalism to obtain predictions of the time required for noise-induced hopping between basins of different optima. We compare the theoretical predictions with simulations of large ensembles of networks for simple problems in supervised and unsupervised learning.