RLPR: Extrapolating RLVR to General Domains without Verifiers

By simply replacing the rule-based verifier reward of RL VR with the proposed LLM's intrinsic probability reward, RLPR achieves consistent improvements in both mathematical and general domains, even outperforming strong RL methods driven by model-based verifier reward. V erifier requirements of different methods are listed in parentheses. Reinforcement Learning with V erifiable Rewards (RL VR) demonstrates promising potential in advancing the reasoning capabilities of LLMs. However, its success remains largely confined to mathematical and code domains. This primary limitation stems from the heavy reliance on domain-specific verifiers, which results in prohibitive complexity and limited scalability. To address the challenge, our key observation is that LLM's intrinsic probability of generating a correct free-form answer directly indicates its own evaluation of the reasoning reward (i.e., how well the reasoning process leads to the correct answer). Building on this insight, we propose RLPR, a simple verifier-free framework that extrapolates RL VR to broader general domains. RLPR uses the LLM's own token probability scores for reference answers as the reward signal and maximizes the expected We find that addressing the high variance of this noisy probability reward is crucial to make it work, and propose prob-to-reward and stabilizing methods to ensure a precise and stable reward from LLM intrinsic probabilities. Comprehensive experiments in four general-domain benchmarks and three mathematical benchmarks show that RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models. Notably, RLPR outperforms concurrent V eriFree by 7.6 points on TheoremQA and 7.5 points on Minerva, and even surpasses strong verifier-model-dependent approaches General-Reasoner by 1.6 average points across seven benchmarks. Large-scale Reinforcement Learning with V erifiable Rewards (RL VR) has emerged as a promising paradigm to advance the reasoning capabilities of Large Language Models (LLMs) (Jaech et al., 2024; DeepSeek-AI et al., 2025; Hu et al., 2025b; Luo et al., 2025a).