There is growing consensus around the view that aligned and beneficial AI requires a reframing of objectives as being contingent, uncertain, and learnable via interaction with humans [35].

This work was supported in part by the National Science Foundation under grant CCF-2149588 and Cisco, Inc.

Maximising a cumulative reward function that is Markov and stationary, i.e., defined over state-action pairs and independent of time, is sufficient to capture many

Generated text may contain offensive or toxic language, contain significant repetition, orbeofadifferent sentiment than desired by the user. We consider thetaskofunlearningthese misalignments byfine-tuning thelanguage model on signals of whatnot to do.

The pairs () intodisentangled) are decomposed) while error ().