This paper presents an extension of the safe interruptibility (SInt) framework to the multi-agent case. The authors argue that the original definition of safe interruptibility is difficult to use in this case and give a more constrained/informed one called'dynamic safe interruptibility' (DSInt) based on whether the update rule depends on the interruption probability. The joint action case is considered first and it is shown that DSInt can be achieved. The case of independent learners is then considered, with a first result showing that independent Q-learners do not satisfy the conditions of the definition of DSInt. The authors finally propose a model where the agents are aware of each others interruptions, and interrupted observations are pruned from the sequence, and claim that this model verify the definition of DSInt.

Disclaimer: my background is in control theory and only recently I have invested most of time in reading and doing research in the area of machine learning and reinforcement learning with specific focus on robotics and control. I went through the submitted paper carefully, including the supplementary material. Therefore I am quite confident with my assessment, especially since the problem that the addressed problem is well inside my core expertise (adaptive control). As I previously said, I am very confident with the problem, less confident with the theoretical framework (reinforcement learning) used to solve it. The math presented in the paper is relatively shallow and carefully checked.

The authors describe an RL architecture comprised of reward shaping plus representation learning that is used to solve an active classification problem, framed as "diagnosis." In this setting, an agent can measure the value of "symptoms" at some cost, and eventually makes a prediction of what disease is present. The architecture is intended to take advantage of the property that symptoms are sparse but correlated. Reward shaping is used to help the agent learn to quickly find symptoms that are present, while the correlations are used to avoid having the agent measure symptoms that are already "known" based on already-measured ones with high certainty. Experimental results demonstrate a substantial improvement over prior work.

My understanding of the paper: This paper describes a novel algorithm for safe model-based control of a unknown system. This is an important problem space, and I am happy to see new contributions. The proposed approach uses a learnt model of the system, and constrains the policy to avoid actions that could bring the system to un-safe states. Additionally, both policy actions and exploratory actions are affected by this safety constraint. The proposed algorithm is based on reasonable assumptions of Lipschitz continuity within the system dynamics as well as the presence of a Lyapunov function, which provides some quantification of risk-related cost of a state.

Two heuristic mechanisms from neuroevolution study have been imported into the recently proposed evolution strategy for deep reinforcement learning. One is Novelty Search (NS), which aims to bias the search to have more exploration. It try to explore previously unvisited areas in the space of behavior, not in the space of policy parameters. The other is to maintain multiple populations in a single run. The authors proposed three variation of the evolution strategy combining these mechanisms.

While MPC enables nonlinear feedback control by solving an optimal control problem at each timestep, the computational burden tends to be significantly large, making it difficult to optimize a policy within the control period. To address this issue, one possible approach is to utilize terminal value learning to reduce computational costs. However, the learned value cannot be used for other tasks in situations where the task dynamically changes in the original MPC setup. In this study, we develop an MPC framework with goal-conditioned terminal value learning to achieve multitask policy optimization while reducing computational time. Furthermore, by using a hierarchical control structure that allows the upper-level trajectory planner to output appropriate goal-conditioned trajectories, we demonstrate that a robot model is able to generate diverse motions. We evaluate the proposed method on a bipedal inverted pendulum robot model and confirm that combining goal-conditioned terminal value learning with an upper-level trajectory planner enables real-time control; thus, the robot successfully tracks a target trajectory on sloped terrain.

Coordination among connected and autonomous vehicles (CAVs) is advancing due to developments in control and communication technologies. However, much of the current work is based on oversimplified and unrealistic task-specific assumptions, which may introduce vulnerabilities. This is critical because CAVs not only interact with their environment but are also integral parts of it. Insufficient exploration can result in policies that carry latent risks, highlighting the need for methods that explore the environment both extensively and efficiently. This work introduces OPTIMA, a novel distributed reinforcement learning framework for cooperative autonomous vehicle tasks. OPTIMA alternates between thorough data sampling from environmental interactions and multi-agent reinforcement learning algorithms to optimize CAV cooperation, emphasizing both safety and efficiency. Our goal is to improve the generality and performance of CAVs in highly complex and crowded scenarios. Furthermore, the industrial-scale distributed training system easily adapts to different algorithms, reward functions, and strategies.

Multilingual NMT is a viable solution for translating low-resource languages (LRLs) when data from high-resource languages (HRLs) from the same language family is available. However, the training schedule, i.e. the order of presentation of languages, has an impact on the quality of such systems. Here, in a many-to-one translation setting, we propose to apply two algorithms that use reinforcement learning to optimize the training schedule of NMT: (1) Teacher-Student Curriculum Learning and (2) Deep Q Network. The former uses an exponentially smoothed estimate of the returns of each action based on the loss on monolingual or multilingual development subsets, while the latter estimates rewards using an additional neural network trained from the history of actions selected in different states of the system, together with the rewards received. On a 8-to-1 translation dataset with LRLs and HRLs, our second method improves BLEU and COMET scores with respect to both random selection of monolingual batches and shuffled multilingual batches, by adjusting the number of presentations of LRL vs. HRL batches.

The workshop 'AI-based Planning for Cyber-Physical Systems', which took place on February 26, 2024, as part of the 38th Annual AAAI Conference on Artificial Intelligence in Vancouver, Canada, brought together researchers to discuss recent advances in AI planning methods for Cyber-Physical Systems (CPS). CPS pose a major challenge due to their complexity and data-intensive nature, which often exceeds the capabilities of traditional planning algorithms. The workshop highlighted new approaches such as neuro-symbolic architectures, large language models (LLMs), deep reinforcement learning and advances in symbolic planning. These techniques are promising when it comes to managing the complexity of CPS and have potential for real-world applications.

Safe reinforcement learning (RL) is a promising approach for many real-world decision-making problems where ensuring safety is a critical necessity. In safe RL research, while expected cumulative safety constraints (ECSCs) are typically the first choices, chance constraints are often more pragmatic for incorporating safety under uncertainties. This paper proposes a \textit{flipping-based policy} for Chance-Constrained Markov Decision Processes (CCMDPs). The flipping-based policy selects the next action by tossing a potentially distorted coin between two action candidates. The probability of the flip and the two action candidates vary depending on the state. We establish a Bellman equation for CCMDPs and further prove the existence of a flipping-based policy within the optimal solution sets. Since solving the problem with joint chance constraints is challenging in practice, we then prove that joint chance constraints can be approximated into Expected Cumulative Safety Constraints (ECSCs) and that there exists a flipping-based policy in the optimal solution sets for constrained MDPs with ECSCs. As a specific instance of practical implementations, we present a framework for adapting constrained policy optimization to train a flipping-based policy. This framework can be applied to other safe RL algorithms. We demonstrate that the flipping-based policy can improve the performance of the existing safe RL algorithms under the same limits of safety constraints on Safety Gym benchmarks.