Self-Generating Knowledge: The Emergence of Autodidactic AI Models

In December 2022 a team of researchers from the Johns Hopkins University has published a remarkable paper: SELF-INSTRUCT: Aligning Language Model with Self Generated Instructions. It describes a method that serves as an innovative approach to instruction-tuning of pretrained language models (LMs) by utilizing instructional signals from the model itself. Informally, it describes a pipeline that allows a Language Model to generate data that will serve to train the model itself. In a few words, the Self-instruct process is an iterative bootstrapping algorithm. It commences with a seed set of manually-written instructions, which is relatively small (175 in the paper). These instructions form the foundation for guiding the overall generation.