It Helps to Take a Second Opinion: Teaching Smaller LLMs to Deliberate Mutually via Selective Rationale Optimisation

Very large language models (LLMs) such as GPT-4 have shown the ability to handle complex tasks by generating and self-refining step-by-step rationales. Smaller language models (SLMs), typically with < 13B parameters, have been improved by using the data generated from very-large LMs through knowledge distillation. Limited success has been achieved at improving the ability of an SLM to explore the space of possible rationales and evaluate them by itself through self-deliberation. To address this, we propose COALITION, a trainable framework that facilitates interaction between two variants of the same SLM and trains them to generate and refine rationales optimized for the end-task. The variants exhibit different behaviors to produce a set of diverse candidate rationales during the generation and refinement steps. The model is then trained via Selective Rationale Optimization (SRO) to prefer generating rationale candidates that maximize the likelihood of producing the ground-truth answer. During inference, COALITION employs a controller to select the suitable variant for generating and refining the rationales. On five different datasets covering mathematical problems, commonsense reasoning, and natural language inference, COALITION outperforms several baselines by up to 5%. Our ablation studies reveal that cross-communication between the two variants performs better than using the single model to self-refine the rationales. We also demonstrate the applicability of COALITION for LMs of varying scales (4B to 14B parameters) and model families (Mistral, Llama, Qwen, Phi). We release the code for this work here. Modern large language models (LLMs) with hundreds of billions of parameters, such as GPT-4 (Achiam et al., 2023) and PaLM-540B (Chowdhery et al., 2022) have shown a remarkable ability to solve complex tasks by generating step-by-step rationales (Wei et al., 2022a;b; Kojima et al., 2022) and refining them through self-correction (Wang et al., 2023b; Welleck et al., 2023). The ability to think step-by-step becomes more prominent with scale, while smaller language models (SLMs), typically 13B, struggle to generate good quality rationales (Valmeekam et al., 2022; Weng et al., 2023). However, owing to the advantages of SLMs such as lesser costs, latency, and compute requirements, significant efforts have been made to improve their ability to handle complex tasks by using feedback obtained through interactions with LLMs (Tunstall et al., 2023; Hsieh et al., 2023; Gou et al., 2024; Wang et al., 2024b). For instance, OpenAI's usage terms prohibit using GPT-generated outputs to train other models for commercial use. Consequently, efforts have been made to improve SLM performance without reliance on an external teacher LLM. The sample is fed to a controller (step 1) to select the variant (steps 2-3) that generates a rationale (step 4).