Appendices

Note that this safe RL problem is less general than the standard formulation of safe RL. The authors introduce a teacher-student hierarchy. To learn the teacher's policy the following constraints are followed: a1 The unsafe set is contained in the intervention set D D The teacher learns when to intervene and to switch between different interventions. A1.2 RL with probability one constraints We have introduced the safety state to the environment as follows: s First, we discuss our design for the PI controller and discuss the necessary parts for it. The proportional part delivers brute force control by having a large control magnitude for large errors, but it is not effective if the instantaneous error values become small.