Abstract: One key challenge for multi-task Reinforcement learning (RL) in practice is the absence of task indicators. Robust RL has been applied to deal with task ambiguity, but may result in over-conservative policies. To balance the worst-case (robustness) and average performance, we propose Group Distributionally Robust Markov Decision Process (GDR-MDP), a flexible hierarchical MDP formulation that encodes task groups via a latent mixture model. GDR-MDP identifies the optimal policy that maximizes the expected return under the worst-possible qualified belief over task groups within an ambiguity set. We rigorously show that GDR-MDP's hierarchical structure improves distributional robustness by adding regularization to the worst possible outcomes.

Abstract: With the increasing presence of robotic systems and human-robot environments in today's so- ciety, understanding the reasoning behind ac- tions taken by a robot is becoming more im- portant. To increase this understanding, users are provided with explanations as to why a spe- cific action was taken. Among other effects, these explanations improve the trust of users in their robotic partners. One option for creat- ing these explanations is an introspection-based approach which can be used in conjunction with reinforcement learning agents to provide probabilities of success. These can in turn be used to reason about the actions taken by the agent in a human-understandable fashion.

TL;DR: Risk-aversion is essential in many RL applications (e.g., driving, robotic surgery and finance). Some modified RL frameworks consider risk (e.g., by optimizing a risk-measure of the return instead of its expectation), but pose new algorithmic challenges. Instead, it is often suggested to stick with the old and good RL framework, and just set the rewards such that negative outcomes are amplified. Unfortunately, as discussed below, modeling risk using expectation over redefined rewards is often unnatural, impractical or even mathematically impossible, hence cannot replace explicit optimization of risk-measures. This is consistent with similar results from decision theory, where risk optimization is not equivalent to expected utility maximization.

Before diving into the different types of Reinforcement Learning and Algorithms, we should familiarize ourselves with the components of Reinforcement Learning. In the example of a baby exploring the world, the baby (agent) is in the real world (environment) and can be crying, feeling happy, or hungry (state). The baby can therefore choose to eat or sleep (action) and the baby is fulfilled if the baby gets to eat when he/she is hungry (reward). As mentioned at the start of the article, Reinforcement Learning involves exploration, and the output of Reinforcement Learning is an optimal policy. A policy describes the action to take at every state; akin to an instruction manual.

Living creatures and machines interact with the world through their morphology and motions. Recent advances in creating bio-inspired morphing robots and machines have led to the study of variable geometry truss (VGT), structures that can approximate arbitrary geometries and has large degree of freedom to deform. However, they are limited to simple geometries and motions due to the excessively complex control system. While a recent work PneuMesh solves this challenge with a novel VGT design that introduces a selective channel connection strategy, it imposes new challenge in identifying effective channel groupings and control methods. Building on top of the hardware concept presented in PneuMesh, we frame the challenge into a co-design problem and introduce a learning-based model to find a sub-optimal design. Specifically, given an initial truss structure provided by a human designer, we first adopt a genetic algorithm (GA) to optimize the channel grouping, and then couple GA with reinforcement learning (RL) for the control. The model is tailored to the PneuMesh system with customized initialization, mutation and selection functions, as well as the customized translation-invariant state vector for reinforcement learning. The result shows that our method enables a robotic table-based VGT to achieve various motions with a limited number of control inputs. The table is trained to move, lower its body or tilt its tabletop to accommodate multiple use cases such as benefiting kids and painters to use it in different shape states, allowing inclusive and adaptive design through morphing trusses.

Recent works on fake news detection have shown the efficacy of using emotions as a feature for improved performance. However, the cross-domain impact of emotion-guided features for fake news detection still remains an open problem. In this work, we propose an emotion-guided, domain-adaptive, multi-task approach for cross-domain fake news detection, proving the efficacy of emotion-guided models in cross-domain settings for various datasets.

It has been a long-standing dream to design artificial agents that explore their environment efficiently via intrinsic motivation, similar to how children perform curious free play. Despite recent advances in intrinsically motivated reinforcement learning (RL), sample-efficient exploration in object manipulation scenarios remains a significant challenge as most of the relevant information lies in the sparse agent-object and object-object interactions. In this paper, we propose to use structured world models to incorporate relational inductive biases in the control loop to achieve sample-efficient and interaction-rich exploration in compositional multi-object environments. By planning for future novelty inside structured world models, our method generates free-play behavior that starts to interact with objects early on and develops more complex behavior over time. Instead of using models only to compute intrinsic rewards, as commonly done, our method showcases that the self-reinforcing cycle between good models and good exploration also opens up another avenue: zero-shot generalization to downstream tasks via model-based planning. After the entirely intrinsic task-agnostic exploration phase, our method solves challenging downstream tasks such as stacking, flipping, pick & place, and throwing that generalizes to unseen numbers and arrangements of objects without any additional training.

Recent studies in using deep reinforcement learning (DRL) to solve Job-shop scheduling problems (JSSP) focus on construction heuristics. However, their performance is still far from optimality, mainly because the underlying graph representation scheme is unsuitable for modeling partial solutions at each construction step. This paper proposes a novel DRL-based method to learn improvement heuristics for JSSP, where graph representation is employed to encode complete solutions. We design a Graph Neural Network based representation scheme, consisting of two modules to effectively capture the information of dynamic topology and different types of nodes in graphs encountered during the improvement process. To speed up solution evaluation during improvement, we design a novel message-passing mechanism that can evaluate multiple solutions simultaneously. Extensive experiments on classic benchmarks show that the improvement policy learned by our method outperforms state-of-the-art DRL-based methods by a large margin.

This work presents an RL-based agent for outpatient hysteroscopy training. Hysteroscopy is a gynecological procedure for examination of the uterine cavity. Recent advancements enabled performing this type of intervention in the outpatient setup without anaesthesia. While being beneficial to the patient, this approach introduces new challenges for clinicians, who should take additional measures to maintain the level of patient comfort and prevent tissue damage. Our prior work has presented a platform for hysteroscopic training with the focus on the passage of the cervical canal. With this work, we aim to extend the functionality of the platform by designing a subsystem that autonomously performs the task of the passage of the cervical canal. This feature can later be used as a virtual instructor to provide educational cues for trainees and assess their performance. The developed algorithm is based on the soft actor critic approach to smooth the learning curve of the agent and ensure uniform exploration of the workspace. The designed algorithm was tested against the performance of five clinicians. Overall, the algorithm demonstrated high efficiency and reliability, succeeding in 98% of trials and outperforming the expert group in three out of four measured metrics.

Abstract: One of the grand challenges of reinforcement learning is the ability to generalize to new tasks. However, general agents require a set of rich, diverse tasks to train on. Designing a'foundation environment' for such tasks is tricky -- the ideal envi- ronment would support a range of emergent phenomena, an expressive task space, and fast runtime. To take a step towards addressing this research bottleneck, this work presents Powderworld, a lightweight yet expressive simulation environment running directly on the GPU. Within Powderworld, two motivating challenges distributions are presented, one for world-modelling and one for reinforcement learning.