We consider how to most efficiently leverage teleoperator time to collect data for learning robust image-based value functions and policies for sparse reward robotic tasks. To accomplish this goal, we modify the process of data collection to include more than just successful demonstrations of the desired task. Instead we develop a novel protocol that we call Visual Backtracking Teleoperation (VBT), which deliberately collects a dataset of visually similar failures, recoveries, and successes. VBT data collection is particularly useful for efficiently learning accurate value functions from small datasets of image-based observations. We demonstrate VBT on a real robot to perform continuous control from image observations for the deformable manipulation task of T-shirt grasping. We find that by adjusting the data collection process we improve the quality of both the learned value functions and policies over a variety of baseline methods for data collection. Specifically, we find that offline reinforcement learning on VBT data outperforms standard behavior cloning on successful demonstration data by 13% when both methods are given equal-sized datasets of 60 minutes of data from the real robot.

Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sample efficiency. We propose a novel algorithm, Query The Agent (QTA), which significantly improves sample efficiency by estimating the agent's epistemic uncertainty throughout the state space and setting goals in highly uncertain areas. Encouraging the agent to collect data in highly uncertain states allows the agent to improve its estimation of the value function rapidly. QTA utilizes a novel technique for estimating epistemic uncertainty, Predictive Uncertainty Networks (PUN), to allow QTA to assess the agent's uncertainty in all previously observed states. We demonstrate that QTA offers decisive sample efficiency improvements over preexisting methods. Deep reinforcement learning has been demonstrated to be highly effective in a diverse array of sequential decision-making tasks (Silver et al., 2016; Berner et al., 2019). However, deep reinforcement learning remains challenging to implement in the real world, in part because of the massive amount of data required for training.

Mechanical ventilation is a key form of life support for patients with pulmonary impairment. Healthcare workers are required to continuously adjust ventilator settings for each patient, a challenging and time consuming task. Hence, it would be beneficial to develop an automated decision support tool to optimize ventilation treatment. We present DeepVent, a Conservative Q-Learning (CQL) based offline Deep Reinforcement Learning (DRL) agent that learns to predict the optimal ventilator parameters for a patient to promote 90 day survival. We design a clinically relevant intermediate reward that encourages continuous improvement of the patient vitals as well as addresses the challenge of sparse reward in RL. We find that DeepVent recommends ventilation parameters within safe ranges, as outlined in recent clinical trials. The CQL algorithm offers additional safety by mitigating the overestimation of the value estimates of out-of-distribution states/actions. We evaluate our agent using Fitted Q Evaluation (FQE) and demonstrate that it outperforms physicians from the MIMIC-III dataset.

This research studies the network traffic signal control problem. It uses the Lyapunov control function to derive the back pressure method, which is equal to differential queue lengths weighted by intersection lane flows. Lyapunov control theory is a platform that unifies several current theories for intersection signal control. We further use the theorem to derive the flow-based and other pressure-based signal control algorithms. For example, the Dynamic, Optimal, Real-time Algorithm for Signals (DORAS) algorithm may be derived by defining the Lyapunov function as the sum of queue length. The study then utilizes the back pressure as a reward in the reinforcement learning (RL) based network signal control, whose agent is trained with double Deep Q-Network (Double-DQN). The proposed algorithm is compared with several traditional and RL-based methods under passenger traffic flow and mixed flow with freight traffic, respectively. The numerical tests are conducted on a single corridor and on a local grid network under three traffic demand scenarios of low, medium, and heavy traffic, respectively. The numerical simulation demonstrates that the proposed algorithm outperforms the others in terms of the average vehicle waiting time on the network.

Despite the impressive successes of modern reinforcement learning (RL) (Sutton & Barto, 2018) methods in designing efficient specialized control policies for a wide variety of difficult tasks, many studies have highlighted the limited ability of RL agents to generalize to variations of such tasks that would appear easy to a human being (Farebrother et al., 2018; Packer et al., 2018; Zhang et al., 2018; Song et al., 2020; Cobbe et al., 2019). This work is motivated by the idea that networks trained for specific tasks build state representations that can easily be fooled by the ambiguity between observation variables. For instance, in some platform video games, it is possible to design an optimal policy for a given level based solely on background features and progression indicators, rather than on the position of platforms and enemies (Song et al., 2020). While very efficient on this specific level, such a policy might not perform well on another. Conversely, we formulate and evaluate the hypothesis that a network trained to imitate several such specialized policies on a limited set of task variations induces a state representation that lifts the ambiguity and filters out confounding observation variables. Specifically, we investigate whether the process of network distillation (Hinton et al., 2015; Rusu et al., 2016a), inspired by knowledge consolidation in cognitive systems (Wilson & McNaughton, 1994; Ashworth et al., 2014; McClelland et al., 1995), induces valuable state representations. The interplay between distillation and state representation appears to have received little attention so far. We endeavor to fill this gap and investigate how neural distillation can act as a state representation bottleneck in RL. Our contributions are as follows: We propose a generic experimental protocol to evaluate the effects of imitation (via distillation) on state representation.

The Arcade Learning Environment (ALE) has become an essential benchmark for assessing the performance of reinforcement learning algorithms. However, the computational cost of generating results on the entire 57-game dataset limits ALE's use and makes the reproducibility of many results infeasible. We propose a novel solution to this problem in the form of a principled methodology for selecting small but representative subsets of environments within a benchmark suite. We applied our method to identify a subset of five ALE games, called Atari-5, which produces 57-game median score estimates within 10% of their true values. Extending the subset to 10-games recovers 80% of the variance for log-scores for all games within the 57-game set. We show this level of compression is possible due to a high degree of correlation between many of the games in ALE.

Traditional visual navigation methods of micro aerial vehicle (MAV) usually calculate a passable path that satisfies the constraints depending on a prior map. However, these methods have issues such as high demand for computing resources and poor robustness in face of unfamiliar environments. Aiming to solve the above problems, we propose a neuromorphic reinforcement learning method (Neuro-Planner) that combines spiking neural network (SNN) and deep reinforcement learning (DRL) to realize MAV 3D visual navigation with depth camera. Specifically, we design spiking actor network based on two-state LIF (TS-LIF) neurons and its encoding-decoding schemes for efficient inference. Then our improved hybrid deep deterministic policy gradient (HDDPG) and TS-LIF-based spatio-temporal back propagation (STBP) algorithms are used as the training framework for actor-critic network architecture. To verify the effectiveness of the proposed Neuro-Planner, we carry out detailed comparison experiments with various SNN training algorithm (STBP, BPTT and SLAYER) in the software-in-the-loop (SITL) simulation framework. The navigation success rate of our HDDPG-STBP is 4.3\% and 5.3\% higher than that of the original DDPG in the two evaluation environments. To the best of our knowledge, this is the first work combining neuromorphic computing and deep reinforcement learning for MAV 3D visual navigation task.

While recent camera-only 3D detection methods leverage multiple timesteps, the limited history they use significantly hampers the extent to which temporal fusion can improve object perception. Observing that existing works' fusion of multi-frame images are instances of temporal stereo matching, we find that performance is hindered by the interplay between 1) the low granularity of matching resolution and 2) the sub-optimal multi-view setup produced by limited history usage. Our theoretical and empirical analysis demonstrates that the optimal temporal difference between views varies significantly for different pixels and depths, making it necessary to fuse many timesteps over long-term history. Building on our investigation, we propose to generate a cost volume from a long history of image observations, compensating for the coarse but efficient matching resolution with a more optimal multi-view matching setup. Further, we augment the per-frame monocular depth predictions used for long-term, coarse matching with short-term, fine-grained matching and find that long and short term temporal fusion are highly complementary. While maintaining high efficiency, our framework sets new state-of-the-art on nuScenes, achieving first place on the test set and outperforming previous best art by 5.2% mAP and 3.7% NDS on the validation set. Code will be released $\href{https://github.com/Divadi/SOLOFusion}{here.}$

An important use case is offered by vehicular ground users are static and have known locations. The same wireless networks, in which UABSs serve as relays authors in [28] extended their previous work by considering between vehicular users and the network, enabling the users multiple UABSs. Unlike these previous works, in this paper, to upload data collected by on-board sensors [5]-[11]. Such we consider traffic conditions characterized by vehicular users user-generated data are collected by the network, and then with a priori unknown locations and we move beyond conventional forwarded to other vehicles by means of BSs or road side meta-RL by accounting for the constraint that simulators units (RSUs). Being able to offer stronger, possibly line-ofsight for previous traffic configurations cannot be revisited. The (LoS), links to vehicles as compared to (static) ground rest of the paper is organized as follows. The system model BSs, UABSs can support demanding vehicle-to-everything and the problem formulation are described in Section II.

Reinforcement Learning is a subfield of Machine Learning in which an agent explores an environment to learn how to perform specific tasks by taking actions with a good outcome and avoiding those with a bad one. A reinforcement learning model will learn from its experiences and will identify which actions lead to the best rewards. In reinforcement learning, the agent takes action based on the state of the environment, and the environment will return the reward and the next state. The agent employs a trial and error method to learn. It initially takes random actions and identifies which actions lead to long-term rewards over time.