While policy-based reinforcement learning (RL) achieves tremendous successes in practice, it is significantly less understood in theory, especially compared with value-based RL. In particular, it remains elusive how to design a provably efficient policy optimization algorithm that incorporates exploration. To bridge such a gap, this paper proposes an Optimistic variant of the Proximal Policy Optimization algorithm (OPPO), which follows an "optimistic version" of the policy gradient direction. This paper proves that, in the problem of episodic Markov decision process with linear function approximation, unknown transition, and adversarial reward with full-information feedback, OPPO achieves O (\sqrt{d 3 H 3 T}) regret. Here d is the feature dimension, H is the episode horizon, and T is the total number of steps.

Portfolio management is the act of making decisions to allocate your funds to a collection of assets for optimal dollar results. When those assets are cryptocurrencies the question is that of allocating funds to digital assets in order to maximise some crypto investment goal, for example, accumulate Bitcoin. In this paper, a reinforcement machine learning approach is built using historical data from the crypto exchange website Polonix with the goal of optimising investor gains over a set period. This model is then benchmarked against standard portfolio strategies used by traders such as buy and hold. The results show that the reinforcement learning approach is extremely effective as an investment optimisation strategy; but the authors warn that historical data is not always a valid way to predict the market.

There is a huge push among the researchers here for accountability. I was presenting a poster on "Objective Mismatch in Model-based Reinforcement Learning" at the Deep RL Workshop, and the crowd was very receptive to the idea that some of our underlying assumptions of how RL works may be flawed. I also happened to be presenting my poster next to a researcher at Google pushing for more metrics of reliability in RL algorithms. This means: how consistent is the performance papers propose when they claim a new "state-of-the-art" across environments and random seeds. This realistic robustness may be the key to getting these algorithms to be more useful on real applications (such as robotics which I will always bring up as a great interpretable platform for RL).

The AlphaZero algorithm for the learning of strategy games via self-play, which has produced superhuman ability in the games of Go, chess, and shogi, uses a quantitative reward function for game outcomes, requiring the users of the algorithm to explicitly balance different components of the reward against each other, such as the game winner and margin of victory. We present a modification to the AlphaZero algorithm that requires only a total ordering over game outcomes, obviating the need to perform any quantitative balancing of reward components. We demonstrate that this system learns optimal play in a comparable amount of time to AlphaZero on a sample game.

Many real-world applications involve teams of agents that have to coordinate their actions to reach a common goal against potential adversaries. This paper focuses on zero-sum games where a team of players faces an opponent, as is the case, for example, in Bridge, collusion in poker, and collusion in bidding. The possibility for the team members to communicate before gameplay---that is, coordinate their strategies ex ante---makes the use of behavioral strategies unsatisfactory. We introduce Soft Team Actor-Critic (STAC) as a solution to the team's coordination problem that does not require any prior domain knowledge. STAC allows team members to effectively exploit ex ante communication via exogenous signals that are shared among the team. STAC reaches near-optimal coordinated strategies both in perfectly observable and partially observable games, where previous deep RL algorithms fail to reach optimal coordinated behaviors.

Learning from demonstrations is a useful way to transfer a skill from one agent to another. While most imitation learning methods aim to mimic an expert skill by following the demonstration step-by-step, imitating every step in the demonstration often becomes infeasible when the learner and its environment are different from the demonstration. In this paper, we propose a method that can imitate a demonstration composed solely of observations, which may not be reproducible with the current agent. Our method, dubbed selective imitation learning from observations (SILO), selects reachable states in the demonstration and learns how to reach the selected states. Our experiments on both simulated and real robot environments show that our method reliably performs a new task by following a demonstration. Videos and code are available at https://clvrai.com/silo .

We introduce an algorithm for model-based hierarchical reinforcement learning to acquire self-contained transition and reward models suitable for probabilistic planning at multiple levels of abstraction. We call this framework Planning with Abstract Learned Models (PALM). By representing subtasks symbolically using a new formal structure, the lifted abstract Markov decision process (L-AMDP), PALM learns models that are independent and modular. Through our experiments, we show how PALM integrates planning and execution, facilitating a rapid and efficient learning of abstract, hierarchical models. We also demonstrate the increased potential for learned models to be transferred to new and related tasks.

Discussions on the interplay of humans and Artificial Intelligence tend to pose the issue in the language of opposition. However, according to the thinking of evolutionary biologist Richard Dawkins, tools such as AI can be better thought of as part of our extended phenotype. A phenotype refers to the observable characteristic of an organism, and the idea of the extended phenotype is that this should not be limited to biological processes, but include all of the effects that the genes have upon their environment, both internally and externally. We are used to defining ourselves strictly by the space we occupy in the physical world. The numbers of non-human cells that occupy our own body outnumber the number of human cells and vast colonies of bacteria swarm within the interior of our digestive tract. Author Robert Svoboda compares the human to a minority government ruling a primarily non-human population.

On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long-time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems.