A Theoretical Results Consider a rewardless

We first bound the maximum increase. The case for maximum decrease is similar. The auxiliary reward function is learned after it is generated. We train each auxiliary reward function for 1M steps. A careful λ schedule helps induce a successful policy that avoids side effects.Algorithm 1: A Require CB-V AE training epochs T Require AUP penalty λ Require Exploration buffer size k Require Auxiliary model training steps L Require AUP model training steps N Require PPO update function PPO-Update Require CB-V AE update function V AE-Update for Step k = 1,...K do Sample random action a s Act (a) S = s S end for Epoch t = 1,...T do Update-V AE(F,S) end for Step i = 1,...L + N do s Starting state for Step l = 1,...L do a = ψ Common refers to those hyperparameters that are the same for each evaluated condition.