Despite significant advances in long-context reasoning by large language models (LLMs), primarily through Online Reinforcement Learning (RL) methods, these approaches incur substantial computational costs and complexity. In contrast, simpler and more economical Offline RL methods remain underexplored. To address this gap, we investigate the effectiveness of Offline RL methods, specifically Direct Preference Optimization (DPO) and its length-desensitized variant LD-DPO, in enhancing the reasoning capabilities of LLMs. Extensive experiments across multiple reasoning benchmarks demonstrate that these simpler Offline RL methods substantially improve model performance, achieving an average enhancement of 3.3\%, with a particularly notable increase of 10.1\% on the challenging Arena-Hard benchmark. Furthermore, we analyze DPO's sensitivity to output length, emphasizing that increasing reasoning length should align with semantic richness, as indiscriminate lengthening may adversely affect model performance. We provide comprehensive descriptions of our data processing and training methodologies, offering empirical evidence and practical insights for developing more cost-effective Offline RL approaches.

Direct Preference Optimization (DPO) is widely utilized in the Reinforcement Learning from Human Feedback (RLHF) phase to align Large Language Models (LLMs) with human preferences, thereby enhancing both their harmlessness and efficacy. However, it has been observed that DPO tends to over-optimize for verbosity, which can detrimentally affect both performance and user experience. In this paper, we conduct an in-depth theoretical analysis of DPO's optimization objective and reveal a strong correlation between its implicit reward and data length. To address this issue, we propose a length-desensitization improvement method for DPO, termed LD-DPO. The proposed method aims to desensitize DPO to data length by decoupling explicit length preference, which is relatively insignificant, from the other implicit preferences, thereby enabling more effective learning of the intrinsic preferences. We utilized two settings (Base and Instruct) of Llama2-13B, Llama3-8B, and Qwen2-7B for experimental validation on various benchmarks including MT-Bench and AlpacaEval 2. The experimental results indicate that LD-DPO consistently outperforms DPO and other baseline methods, achieving more concise responses with a 10-40% reduction in length compared to DPO. We conducted in-depth experimental analyses to demonstrate that LD-DPO can indeed achieve length desensitization and align the model more closely with human-real preferences. "Brevity is the Soul of Wit." Human preference alignment is crucial to enable large language models (LLMs) to be helpful, honest, and harmless. Among the various methods to achieve effective alignment (Dai et al., 2024; Yuan et al., 2024a), Directed Preference Optimization (DPO) has emerged as a promising technique (Rafailov et al., 2024), giving rise to numerous derivative algorithms (Hong et al., 2024; Chen et al., 2024b; Ethayarajh et al., 2024). DPO eliminates the reliance on online Reward Models (RMs) by reparameterizing the reward function in Reinforcement Learning from Human Feedback (RLHF), thereby implementing a simple and stable offline preference learning paradigm. Among the dimensions of human language preferences, detailedness is one of the most straightforward categories that current alignment algorithms can effortlessly capture, as longer texts tend to be richer in content. However, it has been demonstrated that DPO is susceptible to an over-optimization issue in this particular preference dimension (Xu et al., 2024). As shown in Fig.1, this overemphasis results in models that produce excessively verbose responses, which can compromise their instruction-following and reasoning capabilities (Ding et al., 2023; Yuan et al., 2024b).