AI can not only outperform people in many planning tasks, but also teach them how to plan better. All prior work was conducted in fully observable environments, but the real world is only partially observable. To bridge this gap, we developed the first metareasoning algorithm for discovering resource-rational strategies for human planning in partially observable environments. Moreover, we developed an intelligent tutor teaching the automatically discovered strategy by giving people feedback on how they plan in increasingly more difficult problems. We showed that our strategy discovery method is superior to the state-of-the-art and tested our intelligent tutor in a preregistered training experiment with 330 participants. The experiment showed that people's intuitive strategies for planning in partially observable environments are highly suboptimal, but can be substantially improved by training with our intelligent tutor. This suggests our human-centred tutoring approach can successfully boost human planning in complex, partially observable sequential decision problems.

It is well-known that for sparse linear bandits, when ignori ng the dependency on sparsity which is much smaller than the ambient dimension, t he worst-case mini-max regret is null Θ null dT null where d is the ambient dimension and T is the number of rounds. On the other hand, in the benign setting where ther e is no noise and the action set is the unit sphere, one can use divide-and-con quer to achieve null O (1) regret, which is (nearly) independent of d and T . This bound naturally interpolates the regret bounds for the worst-case constant -variance regime (i.e., σ To achieve this variance-aware regret guarantee, we develop a general framework that converts any variance-aware linear bandit algorithm to a varia nce-aware algorithm for sparse linear bandits in a "black-box" manner. Specifica lly, we take two recent algorithms as black boxes to illustrate that the claimed bou nds indeed hold, where the first algorithm can handle unknown-variance cases and th e second one is more efficient. This paper studies the sparse linear stochastic bandit prob lem, which is a special case of linear stochastic bandits. In linear bandits ( Dani et al., 2008), the agent is facing a sequential decision-making problem lasting for T rounds. Dani et al. ( 2008) proved that the minimax optimal regret for linear bandits is null Θ(d T) when the noises are independent Gaussian random variables with means 0 and variances 1 and both θ In real-world applications such as recommendation systems, only a few features may be relevant despite a large candidate feature space. In other words, the high-dimensional linear regime may actually allow a low-dimensional structure.

In reinforcement learning (RL), experience replay-based sampling techniques play a crucial role in promoting convergence by eliminating spurious correlations. However, widely used methods such as uniform experience replay (UER) and prioritized experience replay (PER) have been shown to have sub-optimal convergence and high seed sensitivity respectively. To address these issues, we propose a novel approach called IntrospectiveExperience Replay (IER) that selectively samples batches of data points prior to surprising events. Our method builds upon the theoretically sound reverse experience replay (RER) technique, which has been shown to reduce bias in the output of Q-learning-type algorithms with linear function approximation. However, this approach is not always practical or reliable when using neural function approximation. Through empirical evaluations, we demonstrate that IER with neural function approximation yields reliable and superior performance compared toUER, PER, and hindsight experience replay (HER) across most tasks.

The co-adaptation of robots has been a long-standing research endeavour with the goal of adapting both body and behaviour of a system for a given task, inspired by the natural evolution of animals. Co-adaptation has the potential to eliminate costly manual hardware engineering as well as improve the performance of systems. The standard approach to co-adaptation is to use a reward function for optimizing behaviour and morphology. However, defining and constructing such reward functions is notoriously difficult and often a significant engineering effort. This paper introduces a new viewpoint on the co-adaptation problem, which we call co-imitation: finding a morphology and a policy that allow an imitator to closely match the behaviour of a demonstrator. To this end we propose a co-imitation methodology for adapting behaviour and morphology by matching state distributions of the demonstrator. Specifically, we focus on the challenging scenario with mismatched state- and action-spaces between both agents. We find that co-imitation increases behaviour similarity across a variety of tasks and settings, and demonstrate co-imitation by transferring human walking, jogging and kicking skills onto a simulated humanoid.

Advancements in reinforcement learning (RL) have been remarkable in recent years. However, the limitations of traditional training methods have become increasingly evident, particularly in meta-RL settings where agents face new, unseen tasks. Conventional training approaches are susceptible to failure in such situations as they need more robustness to adversity. Our proposed adversarial training regime for Multi-Task Reinforcement Learning (MT-RL) addresses the limitations of conventional training methods in RL, especially in meta-RL environments where the agent faces new tasks. The adversarial component challenges the agent, forcing it to improve its decision-making abilities in dynamic and unpredictable situations. This component operates without relying on manual intervention or domain-specific knowledge, making it a highly versatile solution. Experiments conducted in multiple MT-RL environments demonstrate that adversarial training leads to better exploration and a deeper understanding of the environment. The adversarial training regime for MT-RL presents a new perspective on training and development for RL agents and is a valuable contribution to the field.

Large-scale AI systems that combine search and learning have reached super-human levels of performance in game-playing, but have also been shown to fail in surprising ways. The brittleness of such models limits their efficacy and trustworthiness in real-world deployments. In this work, we systematically study one such algorithm, AlphaZero, and identify two phenomena related to the nature of exploration. First, we find evidence of policy-value misalignment -- for many states, AlphaZero's policy and value predictions contradict each other, revealing a tension between accurate move-selection and value estimation in AlphaZero's objective. Further, we find inconsistency within AlphaZero's value function, which causes it to generalize poorly, despite its policy playing an optimal strategy. From these insights we derive VISA-VIS: a novel method that improves policy-value alignment and value robustness in AlphaZero. Experimentally, we show that our method reduces policy-value misalignment by up to 76%, reduces value generalization error by up to 50%, and reduces average value error by up to 55%.

A multi-agent deep reinforcement learning-based framework for traffic shaping. The proposed framework offers a key advantage over existing congestion management strategies which is the ability to mitigate hysteresis phenomena. Unlike existing congestion management strategies that focus on breakdown prevention, the proposed framework is extremely effective after breakdown formation. The proposed framework assumes partial connectivity between the automated vehicles which share information. The framework requires a basic level of autonomy defined by one-dimensional longitudinal control. This framework is primarily built using a centralized training, centralized execution multi-agent deep reinforcement learning approach, where longitudinal control is defined by signals of acceleration or deceleration commands which are then executed by all agents uniformly. The model undertaken for training and testing of the framework is based on the well-known Double Deep Q-Learning algorithm which takes the average state of flow within the traffic stream as the model input and outputs actions in the form of acceleration or deceleration values. We demonstrate the ability of the model to shape the state of traffic, mitigate the negative effects of hysteresis, and even improve traffic flow beyond its original level. This paper also identifies the minimum percentage of CAVs required to successfully shape the traffic under an assumption of uniformly distributed CAVs within the loop system. The framework illustrated in this work doesnt just show the theoretical applicability of reinforcement learning to tackle such challenges but also proposes a realistic solution that only requires partial connectivity and continuous monitoring of the average speed of the system, which can be achieved using readily available sensors that measure the speeds of vehicles in reasonable proximity to the CAVs.

A new'outside-the-box' method of teaching artificial intelligence (AI) models to make decisions could provide hope for finding new therapeutic methods for cancer, according to a new study from the University of Surrey. Computer scientists from Surrey have demonstrated that an open ended - or model-free - deep reinforcement learning method is able to stabilize large datasets (of up to 200 nodes) used in AI models. The approach holds open the prospect of uncovering ways to arrest the development of cancer by predicting the response of cancerous cells to perturbations including drug treatment. There are a heart-breaking number of aggressive cancers out there with little to no information on where they come from, let alone how to categorize their behavior. This is where machine learning can Dr Sotiris Moschoyiannis, corresponding author of the study from the University of Surreyprovide real hope for us all.

We study the problem of reinforcement learning for a task encoded by a reward machine. The task is defined over a set of properties in the environment, called atomic propositions, and represented by Boolean variables. One unrealistic assumption commonly used in the literature is that the truth values of these propositions are accurately known. In real situations, however, these truth values are uncertain since they come from sensors that suffer from imperfections. At the same time, reward machines can be difficult to model explicitly, especially when they encode complicated tasks. We develop a reinforcement-learning algorithm that infers a reward machine that encodes the underlying task while learning how to execute it, despite the uncertainties of the propositions' truth values. In order to address such uncertainties, the algorithm maintains a probabilistic estimate about the truth value of the atomic propositions; it updates this estimate according to new sensory measurements that arrive from the exploration of the environment. Additionally, the algorithm maintains a hypothesis reward machine, which acts as an estimate of the reward machine that encodes the task to be learned. As the agent explores the environment, the algorithm updates the hypothesis reward machine according to the obtained rewards and the estimate of the atomic propositions' truth value. Finally, the algorithm uses a Q-learning procedure for the states of the hypothesis reward machine to determine the policy that accomplishes the task. We prove that the algorithm successfully infers the reward machine and asymptotically learns a policy that accomplishes the respective task.

Strong inductive biases give humans the ability to quickly learn to perform a variety of tasks. Although meta-learning is a method to endow neural networks with useful inductive biases, agents trained by meta-learning may sometimes acquire very different strategies from humans. We show that co-training these agents on predicting representations from natural language task descriptions and programs induced to generate such tasks guides them toward more human-like inductive biases. Human-generated language descriptions and program induction models that add new learned primitives both contain abstract concepts that can compress description length. Co-training on these representations result in more human-like behavior in downstream meta-reinforcement learning agents than less abstract controls (synthetic language descriptions, program induction without learned primitives), suggesting that the abstraction supported by these representations is key.