On the Unexpected Abilities of Large Language Models

After being trained on large text corpora, LLMs are endowed with several abilities. However, they do not necessarily elicit the abilities required to follow the users' instructions (Brown et al., 2020; Rae et al., 2021; Thoppilan et al., 2022). Moreover, since the training corpora include both high-quality and low-quality data, they can generate toxic, bias, and harmful content. In other words, they do not necessarily display helpful, honest, and harmless behaviors (Gehman et al., 2021; Bender, 2021; Bommassani et al., 2021; Weidinger et al., 2021; Tamkin et al., 2021). These problems can be alleviated by fine-tuning them, i.e. by continuing the training process with a relatively small set of additional training data. This data is typically consists of a list of instruction and output texts where the former denotes the human instruction to the model and the latter denotes the desired output to the instruction (Zhang et.