Although Reinforcement Learning (RL) is effective for sequential decision-making problems under uncertainty, it still fails to thrive in real-world systems where risk or safety is a binding constraint. In this paper, we formulate the RL problem with safety constraints as a non-zero-sum game. While deployed with maximum entropy RL, this formulation leads to a safe adversarially guided soft actor-critic framework, called SAAC. In SAAC, the adversary aims to break the safety constraint while the RL agent aims to maximize the constrained value function given the adversary's policy. The safety constraint on the agent's value function manifests only as a repulsion term between the agent's and the adversary's policies. Unlike previous approaches, SAAC can address different safety criteria such as safe exploration, mean-variance risk sensitivity, and CVaR-like coherent risk sensitivity. We illustrate the design of the adversary for these constraints. Then, in each of these variations, we show the agent differentiates itself from the adversary's unsafe actions in addition to learning to solve the task. Finally, for challenging continuous control tasks, we demonstrate that SAAC achieves faster convergence, better efficiency, and fewer failures to satisfy the safety constraints than risk-averse distributional RL and risk-neutral soft actor-critic algorithms.

Learning to evaluate and improve policies is a core problem of Reinforcement Learning (RL). Traditional RL algorithms learn a value function defined for a single policy. A recently explored competitive alternative is to learn a single value function for many policies. Here we combine the actor-critic architecture of Parameter-Based Value Functions and the policy embedding of Policy Evaluation Networks to learn a single value function for evaluating (and thus helping to improve) any policy represented by a deep neural network (NN). The method yields competitive experimental results. In continuous control problems with infinitely many states, our value function minimizes its prediction error by simultaneously learning a small set of `probing states' and a mapping from actions produced in probing states to the policy's return. The method extracts crucial abstract knowledge about the environment in form of very few states sufficient to fully specify the behavior of many policies. A policy improves solely by changing actions in probing states, following the gradient of the value function's predictions. Surprisingly, it is possible to clone the behavior of a near-optimal policy in Swimmer-v3 and Hopper-v3 environments only by knowing how to act in 3 and 5 such learned states, respectively. Remarkably, our value function trained to evaluate NN policies is also invariant to changes of the policy architecture: we show that it allows for zero-shot learning of linear policies competitive with the best policy seen during training. Our code is public.

Goal-conditioned Reinforcement Learning (RL) aims at learning optimal policies, given goals encoded in special command inputs. Here we study goal-conditioned neural nets (NNs) that learn to generate deep NN policies in form of context-specific weight matrices, similar to Fast Weight Programmers and other methods from the 1990s. Using context commands of the form "generate a policy that achieves a desired expected return," our NN generators combine powerful exploration of parameter space with generalization across commands to iteratively find better and better policies. A form of weight-sharing HyperNetworks and policy embeddings scales our method to generate deep NNs. Experiments show how a single learned policy generator can produce policies that achieve any return seen during training. Finally, we evaluate our algorithm on a set of continuous control tasks where it exhibits competitive performance.

These research advances have not gone unnoticed by educators. Many universities have begun offering courses on the subject of deep reinforcement learning. The aim of this book is to provide an overview of the field, at the proper level of detail for a graduate course in artificial intelligence. It covers the complete field, from the basic algorithms of Deep Q-learning, to advanced topics such as multi-agent reinforcement learning and meta learning.

For an algorithm to be termed "efficient", its execution time must be constrained by a polynomial function of the input size. It was realised early on that not all issues could be handled thus rapidly, but it was difficult to determine which ones could and which couldn't. Some so-called NP-hard issues are thought to be impossible to answer in polynomial time. NP-hard stands for non-deterministic polynomial-time hardness. This article will be focused on understanding some NP-hard problems and trying to solve them with Reinforcement Learning. Following are the topics to be covered.

We develop reinforcement learning techniques to enable interaction across multiple agents including AIs and humans, with potential applications from AI-assisted design to autonomous driving. Methodological contexts of the research include deep reinforcement learning, inverse reinforcement learning, hierarchical reinforcement learning as well as multi-agent and multi-objective reinforcement learning. FCAI is working on a new paradigm of AI-assisted design that aims to cooperate with designers by supporting and leveraging the creativity and problem-solving of designers. The challenge for such AI is how to infer designers' goals and then help them without being needlessly disruptive. We use generative user models to reason about designers' goals, reasoning, and capabilities. In this call, FCAI is looking for a postdoctoral scholar or research fellow to join our effort to develop AI-assisted design. Suitable backgrounds include deep reinforcement learning, Bayesian inference, cooperative AI, computational cognitive modelling, and user modelling. Computational rationality is an emerging integrative theory of intelligence in humans and machines (1) with applications in human-computer interaction, cooperative AI, and robotics. The theory assumes that observable human behavior is generated by cognitive mechanisms that are adapted to the structure of not only the environment but also the mind and brain itself (2).

The Workshop Program of the Association for the Advancement of Artificial Intelligence's Thirty-Sixth Conference on Artificial Intelligence was held virtually from February 22 – March 1, 2022. There were thirty-nine workshops in the program: Adversarial Machine Learning and Beyond, AI for Agriculture and Food Systems, AI for Behavior Change, AI for Decision Optimization, AI for Transportation, AI in Financial Services: Adaptiveness, Resilience & Governance, AI to Accelerate Science and Engineering, AI-Based Design and Manufacturing, Artificial Intelligence for Cyber Security, Artificial Intelligence for Education, Artificial Intelligence Safety, Artificial Intelligence with Biased or Scarce Data, Combining Learning and Reasoning: Programming Languages, Formalisms, and Representations, Deep Learning on Graphs: Methods and Applications, DE-FACTIFY: Multi-Modal Fake News and Hate-Speech Detection, Dialog System Technology Challenge, Engineering Dependable and Secure Machine Learning Systems, Explainable Agency in Artificial Intelligence, Graphs and More Complex Structures for Learning and Reasoning, Health Intelligence, Human-Centric Self-Supervised Learning, Information-Theoretic Methods for Casual Inference and Discovery, Information Theory for Deep Learning, Interactive Machine Learning, Knowledge Discovery from Unstructured Data in Financial Services, Learning Network Architecture during Training, Machine Learning for Operations Research, Optimal Transports and Structured Data Modeling, Practical Deep Learning in the Wild, Privacy-Preserving Artificial Intelligence, Reinforcement Learning for Education: Opportunities and Challenges, Reinforcement Learning in Games, Robust Artificial Intelligence System Assurance, Scientific Document Understanding, Self-Supervised Learning for Audio and Speech Processing, Trustable, Verifiable and Auditable Federated Learning, Trustworthy AI for Healthcare, Trustworthy Autonomous Systems Engineering, and Video Transcript Understanding. This report contains summaries of the workshops, which were submitted by most, but not all the workshop chairs.

We introduce DeepNash, an autonomous agent capable of learning to play the imperfect information game Stratego from scratch, up to a human expert level. Stratego is one of the few iconic board games that Artificial Intelligence (AI) has not yet mastered. This popular game has an enormous game tree on the order of $10^{535}$ nodes, i.e., $10^{175}$ times larger than that of Go. It has the additional complexity of requiring decision-making under imperfect information, similar to Texas hold'em poker, which has a significantly smaller game tree (on the order of $10^{164}$ nodes). Decisions in Stratego are made over a large number of discrete actions with no obvious link between action and outcome. Episodes are long, with often hundreds of moves before a player wins, and situations in Stratego can not easily be broken down into manageably-sized sub-problems as in poker. For these reasons, Stratego has been a grand challenge for the field of AI for decades, and existing AI methods barely reach an amateur level of play. DeepNash uses a game-theoretic, model-free deep reinforcement learning method, without search, that learns to master Stratego via self-play. The Regularised Nash Dynamics (R-NaD) algorithm, a key component of DeepNash, converges to an approximate Nash equilibrium, instead of 'cycling' around it, by directly modifying the underlying multi-agent learning dynamics. DeepNash beats existing state-of-the-art AI methods in Stratego and achieved a yearly (2022) and all-time top-3 rank on the Gravon games platform, competing with human expert players.

Abstract-- This paper presents the DeltaZ robot, a centimeter-scale, low-cost, delta-style robot that allows for a broad range of capabilities and robust functionalities. Current technologies allow DeltaZ to be 3D-printed from soft and rigid materials so that it is easy to assemble and maintain, and lowers the barriers to utilize. Functionality of the robot stems from its three translational degrees of freedom and a closed form kinematic solution which makes manipulation problems more intuitive compared to other manipulators. Moreover, the low cost of the robot presents an opportunity to democratize manipulators for a research setting. We also describe how the robot can be used as a reinforcement learning benchmark. Open-source 3D-printable designs and code are available to the public.

I hope that you've come across, from algorithms achieving super-human level performance at Atari 2600 games, beating professional players at GO, Dota 2 and StarCraft II and to algorithms controlling nuclear-fusion reactors. These are the success stories of reinforcement learning algorithms combined with deep learning (Deep Reinforcement learning or DeepRL). Google's DeepMind and OpenAI heavily does research in this area and thinks that DeepRL is the future of AI. Some Researchers even think that RL might be the key to Artificial General Intelligence (AGI). Reinforcement Learning (RL) is one of the paradigms of Machine learning along with supervised and unsupervised learning.