Quantile Advantage Estimation for Entropy-Safe Reasoning

Reinforcement Learning with V erifiable Rewards (RL VR) strengthens LLM reasoning but training often oscillates between entropy collapse and entropy explosion. We trace both hazards to the mean-baseline used in value-free RL (e.g., GRPO & DAPO), which improperly penalizes negative-advantage samples under reward outliers. We propose Quantile Advantage Estimation (QAE), replacing the mean with a group-wise K-quantile baseline. QAE induces a response-level, two-regime gate: on hard queries (p 1 K) it reinforces rare successes, while on easy queries (p > 1 K) it targets remaining failures. Under first-order soft-max updates, we prove two-sided entropy safety, giving lower/upper bounds on one-step entropy change that curb explosion and prevent collapse. Empirically, this minimal modification stabilizes entropy, sparsifies credit assignment (with tuned K, roughly 80% of responses receive zero advantage), and yields sustained pass@1 gains on Qwen3-8B/14B-Base across AIME'24/'25 and AMC'23. Reinforcement Learning with V erifiable Rewards (RL VR) (Lambert et al., 2024; DeepSeek-AI et al., 2025; Y ang et al., 2025a) enhances Large Language Models (LLMs) by rewarding verifiable correctness (Phan et al., 2025; Rein et al., 2023). Y et reward-driven optimization often triggers entropy collapse (Y u et al., 2025; Cui et al., 2025): the policy distribution sharpens prematurely, suppressing exploration and ultimately limiting performance.