SELF: Self-Evolution with Language Feedback

Large Language Models (LLMs) have shown impressive adaptability in various fields, yet the optimal pathway of autonomous model evolution remains underexplored. Drawing inspiration from the self-driven learning process of humans, we introduce SELF (Self-Evolution with Language Feedback), a novel learning framework that empowers LLMs to continually self-improve their abilities. SELF initiates with a meta-skill learning process that equips the LLMs with capabilities for self-feedback and self-refinement. SELF employs language-based feedback for detailed and nuanced evaluations, pinpointing response flaws and suggesting refinements. Subsequently, the model engages in an iterative process of self-evolution: they autonomously generate responses to unlabeled instructions, refine these responses interactively, and use the refined and filtered data for iterative self-training, thereby progressively boosting their capabilities. Moreover, the SELF framework equips the model with the ability to self-refine during inference, leading to further improved response quality. Our experiments on mathematical and general tasks demonstrate that SELF enables the model to continually selfimprove without human intervention. The SELF framework indicates a promising direction for the autonomous evolution of LLMs, transitioning them from passive information receivers to active participants in their development. Large Language Models (LLMs), like ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023), stand at the forefront of the AI revolution, demonstrating versatility across tasks. Despite their evident capabilities, the way towards achieving autonomous development of LLMs is still under-explored. The development of automatically improved LLMs can draw inspiration from human self-driven learning mechanisms. When facing new challenges, humans naturally engage in a learning cycle of initial attempts, introspective feedback, and behavior refinement. This leads to a critical question: "Can LLMs mimic the human learning process, utilizing self-refinement to enhance their inherent capabilities?"