
Generated text may contain offensive or toxic language, contain significant repetition, orbeofadifferent sentiment than desired by the user. We consider thetaskofunlearningthese misalignments byfine-tuning thelanguage model on signals of whatnot to do.