Emergent Deception and Emergent Optimization

Language models exhibit some planning capabilities today, and since the training data contains descriptions of plans and is (partly) generated by plans, better representations of plans would decrease the training loss. Moreover, similar to chain-of-thought, planning is a complex capability that requires multiple steps to "go right" in order to be successful. Thus, planning satisfies both of the principles for emergence described earlier and is a good candidate for future emergent behavior. Since planning could also increase reward hacking, we should be on the lookout for planning capabilities in models and for ways to ameliorate any reward hacking that might occur. The main takeaway is that emergent risks, rather than being an abstract concern, can be concretely predicted in at least some cases.