LLM Routing with Dueling Feedback

We study LLM routing, the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost. We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores, thereby yielding label-efficient and dynamic adaptation. Building on this formulation, we introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting. These embeddings enable the practical instantiation of Feel-Good Thompson Sampling for Contextual Dueling Bandits (FGTS.CDB), a theoretically grounded posterior-sampling algorithm. We propose four variants of the categorical weighting that explicitly integrate model quality and cost, and we empirically evaluate the proposed methods on the RouterBench and MixInstruct datasets. Across both benchmarks, our methods achieve lower cumulative regret and faster convergence, with better robustness and performance-cost balance than strong baselines built with a general-purpose OpenAI embedding model. The potential of large language models (LLMs) is so great that they have become a necessary part of daily life, with applications ranging from office assistance and fashion/dining suggestions to entertainment. LLM routing refers to a problem of dynamically selecting the most suitable LLM from a set of candidates for each query in a sequence of questions. Before the emergence of a universally dominant and affordable foundation model, routing is important because the choice of LLM must align with user traits, model expertise, and cost. To balance these three key factors, cascading algorithms such as FrugalGPT (Chen et al., 2024) and AutoMix (Aggarwal et al., 2024) were first proposed. The idea is to query a cheaper model first and advance the query to a more expensive one if the current response is unlikely to meet the user's expectation. A drawback of cascading is the accumulated cost and latency caused by calling multiple LLM candidates to generate the final response.