Recent advances in high-fidelity simulators have enabled closed-loop training of autonomous driving agents, potentially solving the distribution shift in training v.s. deployment and allowing training to be scaled both safely and cheaply. However, there is a lack of understanding of how to build effective training benchmarks for closed-loop training. In this work, we present the first empirical study which analyzes the effects of different training benchmark designs on the success of learning agents, such as how to design traffic scenarios and scale training environments. Furthermore, we show that many popular RL algorithms cannot achieve satisfactory performance in the context of autonomous driving, as they lack long-term planning and take an extremely long time to train. To address these issues, we propose trajectory value learning (TRAVL), an RL-based driving agent that performs planning with multistep look-ahead and exploits cheaply generated imagined data for efficient learning. Our experiments show that TRAVL can learn much faster and produce safer maneuvers compared to all the baselines. For more information, visit the project website: https://waabi.ai/research/travl

This paper addresses the challenge of industrial bin picking using entangled wire harnesses. Wire harnesses are essential in manufacturing but poses challenges in automation due to their complex geometries and propensity for entanglement. Our previous work tackled this issue by proposing a quasi-static pulling motion to separate the entangled wire harnesses. However, it still lacks sufficiency and generalization to various shapes and structures. In this paper, we deploy a dual-arm robot that can grasp, extract and disentangle wire harnesses from dense clutter using dynamic manipulation. The robot can swing to dynamically discard the entangled objects and regrasp to adjust the undesirable grasp pose. To improve the robustness and accuracy of the system, we leverage a closed-loop framework that uses haptic feedback to detect entanglement in real-time and flexibly adjust system parameters. Our bin picking system achieves an overall success rate of 91.2% in the real-world experiments using two different types of long wire harnesses. It demonstrates the effectiveness of our system in handling various wire harnesses for industrial bin picking.

Automatically assigning tasks to people is challenging because human performance can vary across tasks for many reasons. This challenge is further compounded in real-life settings in which no oracle exists to assess the quality of human decisions and task assignments made. Instead, we find ourselves in a "closed" decision-making loop in which the same fallible human decisions we rely on in practice must also be used to guide task allocation. How can imperfect and potentially biased human decisions train an accurate allocation model? Our key insight is to exploit weak prior information on human-task similarity to bootstrap model training. We show that the use of such a weak prior can improve task allocation accuracy, even when human decision-makers are fallible and biased. We present both theoretical analysis and empirical evaluation over synthetic data and a social media toxicity detection task. Results demonstrate the efficacy of our approach.

This paper presents a closed-loop magnetic manipulation framework for robotic transesophageal echocardiography (TEE) acquisitions. Different from previous work on intracorporeal robotic ultrasound acquisitions that focus on continuum robot control, we first investigate the use of magnetic control methods for more direct, intuitive, and accurate manipulation of the distal tip of the probe. We modify a standard TEE probe by attaching a permanent magnet and an inertial measurement unit sensor to the probe tip and replacing the flexible gastroscope with a soft tether containing only wires for transmitting ultrasound signals, and show that 6-DOF localization and 5-DOF closed-loop control of the probe can be achieved with an external permanent magnet based on the fusion of internal inertial measurement and external magnetic field sensing data. The proposed method does not require complex structures or motions of the actuator and the probe compared with existing magnetic manipulation methods. We have conducted extensive experiments to validate the effectiveness of the framework in terms of localization accuracy, update rate, workspace size, and tracking accuracy. In addition, our results obtained on a realistic cardiac tissue-mimicking phantom show that the proposed framework is applicable in real conditions and can generally meet the requirements for tele-operated TEE acquisitions.

Generally, system operators conduct the economic operation of power systems in an open-loop predict-then-optimize process: the renewable energy source (RES) availability and system reserve requirements are first predicted; given the predictions, system operators solve optimization models such as unit commitment (UC) to determine the economical operation plans accordingly. However, such an open-loop process could essentially compromise the operation economics because its predictors myopically seek to improve the immediate statistical prediction errors instead of the ultimate operation cost. To this end, this paper presents a closed-loop predict-and-optimize framework, offering a prescriptive UC to improve the operation economics. First, a bilevel mixed-integer programming model is leveraged to train cost-oriented predictors tailored for optimal system operations: the upper level trains the RES and reserve predictors based on their induced operation cost; the lower level, with given predictions, mimics the system operation process and feeds the induced operation cost back to the upper level. Furthermore, the embeddability of the trained predictors grants a prescriptive UC model, which simultaneously provides RES-reserve predictions and UC decisions with enhanced operation economics. Finally, numerical case studies using real-world data illustrate the potential economic and practical advantages of prescriptive UC over deterministic, robust, and stochastic UC models.

NIMS-OS (NIMS Orchestration System) is a Python library created to realize a closed loop of robotic experiments and artificial intelligence (AI) without human intervention for automated materials exploration. It uses various combinations of modules to operate autonomously. Each module acts as an AI for materials exploration or a controller for a robotic experiments. As AI techniques, Bayesian optimization (PHYSBO), boundless objective-free exploration (BLOX), phase diagram construction (PDC), and random exploration (RE) methods can be used. Moreover, a system called NIMS automated robotic electrochemical experiments (NAREE) is available as a set of robotic experimental equipment. Visualization tools for the results are also included, which allows users to check the optimization results in real time. Newly created modules for AI and robotic experiments can be added easily to extend the functionality of the system. In addition, we developed a GUI application to control NIMS-OS.To demonstrate the operation of NIMS-OS, we consider an automated exploration for new electrolytes. NIMS-OS is available at https://github.com/nimsos-dev/nimsos.

We present a novel approach to efficiently compute tight non-convex enclosures of the image through neural networks with ReLU, sigmoid, or hyperbolic tangent activation functions. In particular, we abstract the input-output relation of each neuron by a polynomial approximation, which is evaluated in a set-based manner using polynomial zonotopes. While our approach can also can be beneficial for open-loop neural network verification, our main application is reachability analysis of neural network controlled systems, where polynomial zonotopes are able to capture the non-convexity caused by the neural network as well as the system dynamics. This results in a superior performance compared to other methods, as we demonstrate on various benchmarks. Keywords: Neural network verification neural network controlled systems reachability analysis polynomial zonotopes formal verification.

Closed-loop control relies on sensory feedback that is usually as(cid:173) sumed to be free . But if sensing incurs a cost, it may be cost(cid:173) effective to take sequences of actions in open-loop mode. We de(cid:173) scribe a reinforcement learning algorithm that learns to combine open-loop and closed-loop control when sensing incurs a cost. Al(cid:173) though we assume reliable sensors, use of open-loop control means that actions must sometimes be taken when the current state of the controlled system is uncertain. This is a special case of the hidden-state problem in reinforcement learning, and to cope, our algorithm relies on short-term memory.

The Koopman operator allows a nonlinear system to be rewritten as an infinite-dimensional linear system by viewing it in terms of an infinite set of lifting functions instead of a state vector. The main feature of this representation is its linearity, making it compatible with existing linear systems theory. A finite-dimensional approximation of the Koopman operator can be identified from experimental data by choosing a finite subset of lifting functions, applying it to the data, and solving a least squares problem in the lifted space. Existing Koopman operator approximation methods are designed to identify open-loop systems. However, it is impractical or impossible to run experiments on some systems without a feedback controller. Unfortunately, the introduction of feedback control results in correlations between the system's input and output, making some plant dynamics difficult to identify if the controller is neglected. This paper addresses this limitation by introducing a method to identify a Koopman model of the closed-loop system, and then extract a Koopman model of the plant given knowledge of the controller. This is accomplished by leveraging the linearity of the Koopman representation of the system. The proposed approach widens the applicability of Koopman operator identification methods to a broader class of systems. The effectiveness of the proposed closed-loop Koopman operator approximation method is demonstrated experimentally using a Harmonic Drive gearbox exhibiting nonlinear vibrations.

In many specific scenarios, accurate and effective system identification is a commonly encountered challenge in the model predictive control (MPC) formulation. As a consequence, the overall system performance could be significantly weakened in outcome when the traditional MPC algorithm is adopted under those circumstances when such accuracy is lacking. This paper investigates a non-parametric closed-loop behavior learning method for multi-agent motion planning, which underpins a data-driven predictive control framework. Utilizing an innovative methodology with closed-loop input/output measurements of the unknown system, the behavior of the system is learned based on the collected dataset, and thus the constructed non-parametric predictive model can be used to determine the optimal control actions. This non-parametric predictive control framework alleviates the heavy computational burden commonly encountered in the optimization procedures typically in alternate methodologies requiring open-loop input/output measurement data collection and parametric system identification. The proposed data-driven approach is also shown to preserve good robustness properties. Finally, a multi-UAV system is used to demonstrate the highly effective outcome of this promising development.