Provable Safe Reinforcement Learning with Binary Feedback

Reinforcement learning (RL) is an important paradigm that can be used to solve important dynamic decisionmaking problems in a diverse set of fields, such as robotics, transportation, healthcare, and user assistance. In recent years there has been a significant increase in interest in this problem, with many proposed solutions. However, in many such applications there are important safety considerations that are difficult to address with existing techniques. Consider the running example of a cleaning robot, whose task is to learn how to vacuum the floor of a house. The primary goal of the robot, of course, is to learn to vacuum as efficiently as possible, which may be measured by the amount cleaned in a given time. However, we would also like to impose safety constraints on the robot's actions; for example, the robot shouldn't roll off of a staircase where it could damage itself, it shouldn't roll over electrical cords, or it shouldn't vacuum up the owner's possessions. In this example, there are several desirable properties we would like a safety-aware learning algorithm to have, including: 1. The agent should avoid taking any unsafe actions, even during training 2. Since it is hard to concretely define a safety function from the robot's sensory observations a priori, we would like the agent to learn a safety function given feedback of observed states 3. Since the notion of safety is human-defined, and we would like the safety feedback to be manually provided by humans (e.g. the owner), we would want the agent to ask for as little feedback as possible 4. We would like to use binary feedback (i.e. is an action in a given state safe or unsafe) rather than numeric feedback, as this is more natural for humans to provide 5. Since the agent may need to act in real time without direct intervention, they should only ask for feedback offline in between episodes