Reward Model Routing in Alignment

Reinforcement learning from human or AI feedback (RLHF/RLAIF) has become the standard paradigm for aligning large language models (LLMs). However, most pipelines rely on a single reward model (RM), limiting alignment quality and risking overfitting. Recent work explores RM routing--dynamically selecting an RM from a candidate pool to exploit complementary strengths while maintaining O(1) RM calls--but existing methods suffer from cold-start and insufficient exploration. We propose BayesianRouter, a hybrid routing framework that combines offline RM strengths learning with online Bayesian selection. In the offline stage, a multi-task router is trained on preference data to estimate per-RM reliability. In the online stage, a Bayesian Thompson sampling router performs per-query RM selection, initializing RM-specific weight vectors with offline embeddings as Gaussian priors and adaptively updating their posteriors with online rewards to adapt to the evolving policy distribution. Extensive experiments on instruction-following (AlpacaEval-2, Arena-Hard, MT -Bench) and reasoning (GSM8K, MMLU) benchmarks show that BayesianRouter consistently outperforms individual RMs, RM ensembling, and existing routing methods. Large language models (LLMs) have revolutionized artificial intelligence, demonstrating substantial capabilities in language understanding, reasoning, and open-ended text generation across diverse domains (Guo et al., 2025; Achiam et al., 2023). To safely and effectively deploy LLMs, recent post-training techniques, particularly reinforcement learning from human feedback (RLHF) and its AI-augmented variant (RLAIF) (Wang et al., 2024b), aimed at aligning LLMs with human values and preferences. In standard RLHF, a reward model (RM) provides the feedback signal for optimizing the policy LLM; for example, in PPO, the RM provides a scalar reward to directly increase the probability of preferred responses (Ziegler et al., 2019), while in direct preference optimization (DPO), the RM compares two candidate responses to determine which is better (Dong et al., 2024).