Today we focus on building a Monte Carlo (MC) agent to learn a MDP. In a previous story, we implemented a model-based ADP learner which estimates a model of reward function r(s) and transition probabilities p(s′ s, a). This model-based approach may work efficiently in some cases. However, if the transition model is difficult to estimate, a model-free approach tends to be a better choice. Monte Carlo (MC), which is our topic today, is one example of such model-free approaches. The code in this story is part of our MAD from scratch project where MAD stands for machine learning, artificial intelligence, and data science. In model-based methods, our policy is derived from the utility values of states.

A longstanding goal of the field of AI is a strategy for compiling diverse experience into a highly capable, generalist agent. In the subfields of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets. Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents. Specifically, we show that a single transformer-based model - with a single set of weights - trained purely offline can play a suite of up to 46 Atari games simultaneously at close-to-human performance. When trained and evaluated appropriately, we find that the same trends observed in language and vision hold, including scaling of performance with model size and rapid adaptation to new games via fine-tuning. We compare several approaches in this multi-game setting, such as online and offline RL methods and behavioral cloning, and find that our Multi-Game Decision Transformer models offer the best scalability and performance. We release the pre-trained models and code to encourage further research in this direction. Additional information, videos and code can be seen at: sites.google.com/view/multi-game-transformers

We consider a distributed reinforcement learning setting where multiple agents separately explore the environment and communicate their experiences through a central server. However, $\alpha$-fraction of agents are adversarial and can report arbitrary fake information. Critically, these adversarial agents can collude and their fake data can be of any sizes. We desire to robustly identify a near-optimal policy for the underlying Markov decision process in the presence of these adversarial agents. Our main technical contribution is Weighted-Clique, a novel algorithm for the robust mean estimation from batches problem, that can handle arbitrary batch sizes. Building upon this new estimator, in the offline setting, we design a Byzantine-robust distributed pessimistic value iteration algorithm; in the online setting, we design a Byzantine-robust distributed optimistic value iteration algorithm. Both algorithms obtain near-optimal sample complexities and achieve superior robustness guarantee than prior works.

The scientists approached it with a model-free technique, which is also known as deep reinforcement learning.

In this final course, you will put together your knowledge from Courses 1, 2 and 3 to implement a complete RL solution to a problem. This capstone will let you see how each component---problem formulation, algorithm selection, parameter selection and representation design---fits together into a complete solution, and how to make appropriate choices when deploying RL in the real world. This project will require you to implement both the environment to stimulate your problem, and a control agent with Neural Network function approximation. In addition, you will conduct a scientific study of your learning system to develop your ability to assess the robustness of RL agents. To use RL in the real world, it is critical to (a) appropriately formalize the problem as an MDP, (b) select appropriate algorithms, (c) identify what choices in your implementation will have large impacts on performance and (d) validate the expected behaviour of your algorithms.

RL development is being driven by several companies and research groups, including Google, Microsoft, and Facebook. It requires lots of investment in research, as there are not that many directions that are developed enough to be able to just take their methods and apply them to a problem. This is similar to how natural language processing and computer vision were several years ago. Having said that, the field of RL is attracting lots of attention, both from researchers and practitioners. This book helps readers to understand RL methods using real-life problems, and make the exciting RL domain accessible to a much wider audience than just research groups or large AI companies.

Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Embodied AI. Whereas many approaches and previous survey pursuits have characterised one or two of these dimensions, there has not been a holistic analysis at the center of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and opportunities for the field. In this survey paper, we discuss Embodied Vision-Language Planning (EVLP) tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language. We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulated environments, as well as the datasets used for EVLP tasks. Finally, we present the core challenges that we believe new EVLP works should seek to address, and we advocate for task construction that enables model generalizability and furthers real-world deployment.

This is the fourth of a series of articles in which I summarize the lectures from CS285 held by Professor Sergey Levine, to whom all credit goes. All images are taken from his lectures. This article I wrote is an introduction to deep reinforcement learning. Actor-critic algorithms build on the policy gradient framework that we discussed in this article. On top, they're also augmented with learned value functions and Q-functions. What if we just learn a value function and then try to use that value function to figure out how to act? The intuition for why this should be possible is that the value function tells us which states are better than others, so if we simply select actions that go into the better states maybe we don't need an explicit policy neural network anymore. Here's the the way to make this intuition a bit more formal.

We study reinforcement learning (RL) in settings where observations are high-dimensional, but where an RL agent has access to abstract knowledge about the structure of the state space, as is the case, for example, when a robot is tasked to go to a specific room in a building using observations from its own camera, while having access to the floor plan. We formalize this setting as transfer reinforcement learning from an abstract simulator, which we assume is deterministic (such as a simple model of moving around the floor plan), but which is only required to capture the target domain's latent-state dynamics approximately up to unknown (bounded) perturbations (to account for environment stochasticity). Crucially, we assume no prior knowledge about the structure of observations in the target domain except that they can be used to identify the latent states (but the decoding map is unknown). Under these assumptions, we present an algorithm, called TASID, that learns a robust policy in the target domain, with sample complexity that is polynomial in the horizon, and independent of the number of states, which is not possible without access to some prior knowledge. In synthetic experiments, we verify various properties of our algorithm and show that it empirically outperforms transfer RL algorithms that require access to "full simulators" (i.e., those that also simulate observations).

Before starting the article, it is important to understand what an agent in AI is. The agent is basically an entity that helps the AI, machine learning, or deep reinforcement learning to make a decision or trigger the AI to make a decision. In terms of software, it is defined as the entity which can take decisions and can make different decisions on the basis of changes in the environment, or after getting input from the external environment. In simpler words, the quick agent perceives external change and acts against it the better the results obtained from the model. Hence the role of the agent is always very important in artificial intelligence, machine learning, and deep learning.