Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024c) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge and follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision. Large Language Models (LLMs) are advancing significantly in their ability to follow instructions and respond to user queries (OpenAI, 2023; Touvron et al., 2023). An important phase in training these models is instruction tuning (Ouyang et al., 2022), which typically involves training LLMs on datasets curated by humans, either via supervised finetuning or preference optimization. Nevertheless, the acquisition of human-generated data is both costly and time-consuming. Furthermore, the quality of such data is inherently constrained by the limitations of human capabilities. The so-called'Super Alignment' challenge (Burns et al., 2023) aims to find a solution to steering or controlling potentially super-intelligent AIs when their actions are inherently beyond human abilities to judge. Among the potential solutions to this challenge, self-judging by the AI emerges as a particularly promising approach.