Does RLHF Scale? Exploring the Impacts From Data, Model, and Method

This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). Although RLHF is considered an important step in post-training of LLMs, its scaling potential is still largely unknown. We systematically analyze key components in the RLHF framework--model size, data composition, and inference budget--and their impacts on performance. Our findings show that increasing data diversity and volume improves reward model performance, helping process-supervision models scale better. For policy training, more response samples per prompt boost performance initially but quickly plateau. And larger reward models offer modest gains in policy training. In addition, larger policy models benefit less from RLHF with a fixed reward model. Based on these observations, we propose strategies to optimize RLHF performance within computational limits. Large Language Models (LLMs) have revolutionized natural language processing by learning extensive language patterns from vast datasets. A key step in these models is reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022), which helps align the model's behavior with human intentions and enhances their performance across diverse tasks such as text generation Hu et al. (2024), coding (Li et al., 2022; Zhu et al., 2024), and mathematical reasoning (Zhu et al., 2024) before deployment.