Uncertainty-Penalized Direct Preference Optimization

Aligning Large Language Models (LLMs) to human preferences in content, style, and presentation is challenging, in part because preferences are varied, contextdependent, and sometimes inherently ambiguous. While successful, Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are prone to the issue of proxy reward overoptimization. Analysis of the DPO loss reveals a critical need for regularization for mislabeled or ambiguous preference pairs to avoid reward hacking. In this work, we develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes, inspired by offline reinforcement learning. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. Evaluation of the methods is performed with GPT2 Medium on the Anthropic-HH dataset using a model ensemble to obtain uncertainty estimates, and shows improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses. Aligning LLMs to human preferences in content, style, and presentation has become a central challenge in improving and deploying LLMs, leading to the advent of Reinforcement Learning with Human Feedback (RLHF), now a prominent technique to fine-tune state-of-the-art LLMs (Casper et al., 2023). The standard RLHF pipeline involves human feedback collection, reward model training, and LLM policy optimization via reinforcement learning (RL). Despite its success, each stage presents challenges, from feedback interpretation and policy generalization to challenging RL implementation (Casper et al., 2023). Direct Preference Optimisation (DPO) (Rafailov et al., 2023) effectively bypasses the reward model by fine-tuning the policy to maximize the likelihood of the preference data under the Bradley-Terry model (A. DPO is easier to implement than RL algorithms, and benefits from computational efficiency and stability by avoiding potential inaccuracies and biases of a reward model (Xu et al., 2024; Casper et al., 2023).