Reinforcement Learning
Google Accelerates Quantum Computation with Classical Machine Learning
Tech giant Google's recent claim regarding quantum supremacy created a buzz in the computer science community and got global mainstream media talking about quantum computing breakthroughs. Yesterday Google fed the public's growing interest in the topic with a blog post introducing a study on improving quantum computation using classical machine learning. The qubit is the most basic constituent of quantum computing, and also poses one of the most significant challenges for the realization of near-term quantum computers. Various characteristics of qubits have made it challenging to control them. Google AI explains that issues such as imperfections in the control electronics can "impact the fidelity of the computation and thus limit the applications of near-term quantum devices."
Attention-based Fault-tolerant Approach for Multi-agent Reinforcement Learning Systems
Geng, Mingyang, Xu, Kele, Li, Yiying, Liu, Shuqi, Ding, Bo, Wang, Huaimin
The aim of multi-agent reinforcement learning systems is to provide interacting agents with the ability to collaboratively learn and adapt to the behavior of other agents. In many real-world applications, the agents can only acquire a partial view of the world. However, in realistic settings, one or more agents that show arbitrarily faulty or malicious behavior may suffice to let the current coordination mechanisms fail. In this paper, we study a practical scenario considering the security issues in the presence of agents with arbitrarily faulty or malicious behavior. Under these circumstances, learning an optimal policy becomes particularly challenging, even in the unrealistic case that an agent's policy can be made conditional upon all other agents' observations. To overcome these difficulties, we present an Attention-based Fault-Tolerant (FT-Attn) algorithm which selects correct and relevant information for each agent at every time-step. The multi-head attention mechanism enables the agents to learn effective communication policies through experience concurrently to the action policies. Empirical results have shown that FT-Attn beats previous state-of-the-art methods in some complex environments and can adapt to various kinds of noisy environments without tuning the complexity of the algorithm. Furthermore, FT-Attn can effectively deal with the complex situation where an agent needs to reach multiple agents' correct observation at the same time.
End-to-End Motion Planning of Quadrotors Using Deep Reinforcement Learning
Separation of these tasks is the medium within the current state-of- the-art navigation methods. Each task is performed by an individual module and modularity is attained easily by this way. Nevertheless, modularity comes with the cost of possible incompatibility, especially with the presence of erroneous modules. An erroneous module in the pipeline could easily cause the other modules to fail as well. Therefore, in this work, the unification of these tasks is attempted within a single, reliable module using deep reinforcement learning (RL) [13]-[16].
Ready, Set, Algorithms! Teams Learn AI by Racing Cars
The DeepRacer league, developed by Amazon Web Services, is designed to teach a branch of artificial intelligence known as reinforcement learning. Amazon Web Services (AWS) has developed the DeepRacer League, a competition designed to teach a branch of artificial intelligence (AI) known as reinforcement learning, in which algorithms learn the correct way to perform an action based on trial and error, and observations. As part of the DeepRacer League, teams or individuals build and train AI algorithms using Amazon SageMaker software, then deploy them to self-driving model cars measuring about 10 inches long, which they race around a track roughly 17 feet by 26 feet. Morningstar is one of the companies participating in the DeepRacer League, and thanks to the training, the company expects to have dozens of projects based on reinforcement learning and other machine learning techniques in deployment by the end of next year. AWS developed the DeepRacer program in an effort to teach software developers about machine learning in a more engaging way than reading scientific articles.
DeepMind Has Quietly Open Sourced Three New Impressive Reinforcement Learning Frameworks
Deep reinforcement learning(DRL) has been at the center of some of the biggest breakthroughs of artificial intelligence(AI) in the last few years. However, despite all its progress, DRL methods remain incredibly difficult to apply in mainstream solutions given the lack of tooling and libraries. Consequently, DRL remains mostly a research activity that hasn't seen a lot of adoption into real world machine learning solutions. Addressing that problem requires better tools and frameworks. Among the current generation of artificial intelligence(AI) leaders, DeepMind stands alone as the company that has done the most to advance DRL research and development. Recently, the Alphabet subsidiary has been releasing a series of new open source technologies that can help to streamline the adoption of DRL methods.
If MaxEnt RL is the Answer, What is the Question?
Eysenbach, Benjamin, Levine, Sergey
Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not optimize expected utility. In this paper, we formally show that MaxEnt RL does optimally solve certain classes of control problems with variability in the reward function. In particular, we show (1) that MaxEnt RL can be used to solve a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player game where an adversary chooses the reward function. These results suggest a deeper connection between MaxEnt RL, robust control, and POMDPs, and provide insight for the types of problems for which we might expect MaxEnt RL to produce effective solutions. Specifically, our results suggest that domains with uncertainty in the task goal may be especially well-suited for MaxEnt RL methods.
"I'm sorry Dave, I'm afraid I can't do that" Deep Q-learning from forbidden action
Seurin, Mathieu, Preux, Philippe, Pietquin, Olivier
The use of Reinforcement Learning (RL) is still restricted to simulation or to enhance human-operated systems through recommendations. Real-world environments (e.g. industrial robots or power grids) are generally designed with safety constraints in mind implemented in the shape of valid actions masks or contingency controllers. For example, the range of motion and the angles of the motors of a robot can be limited to physical boundaries. Violating constraints thus results in rejected actions or entering in a safe mode driven by an external controller, making RL agents incapable of learning from their mistakes. In this paper, we propose a simple modification of a state-of-the-art deep RL algorithm (DQN), enabling learning from forbidden actions. To do so, the standard Q-learning update is enhanced with an extra safety loss inspired by structured classification. We empirically show that it reduces the number of hit constraints during the learning phase and accelerates convergence to near-optimal policies compared to using standard DQN. Experiments are done on a Visual Grid World Environment and Text-World domain.
Discounted Reinforcement Learning is Not an Optimization Problem
Naik, Abhishek, Shariff, Roshan, Yasui, Niko, Sutton, Richard S.
Discounted reinforcement learning is fundamentally incom patible with function approximation for control in continuing tasks. This is beca use it is not an optimization problem -- it lacks an objective function. After s ubstantiating these claims, we go on to address some misconceptions about discou nting and its connection to the average reward formulation. W e encourage res earchers to adopt rigorous optimization approaches for reinforcement learn ing in continuing tasks, such as average reward.
Manufacturing Dispatching using Reinforcement and Transfer Learning
Zheng, Shuai, Gupta, Chetan, Serita, Susumu
Efficient dispatching rule in manufacturing industry is key to ensure product on-time delivery and minimum past-due and inventory cost. Manufacturing, especially in the developed world, is moving towards on-demand manufacturing meaning a high mix, low volume product mix. This requires efficient dispatching that can work in dynamic and stochastic environments, meaning it allows for quick response to new orders received and can work over a disparate set of shop floor settings. In this paper we address this problem of dispatching in manufacturing. Using reinforcement learning (RL), we propose a new design to formulate the shop floor state as a 2-D matrix, incorporate job slack time into state representation, and design lateness and tardiness rewards function for dispatching purpose. However, maintaining a separate RL model for each production line on a manufacturing shop floor is costly and often infeasible. To address this, we enhance our deep RL model with an approach for dispatching policy transfer. This increases policy generalization and saves time and cost for model training and data collection. Experiments show that: (1) our approach performs the best in terms of total discounted reward and average lateness, tardiness, (2) the proposed policy transfer approach reduces training time and increases policy generalization.