L3Ms -- Lagrange Large Language Models

Supervised fine-tuning (SFT) and alignment of large language models (LLMs) are key steps in providing a good user experience. However, the concept of an appropriate alignment is inherently application-dependent, and current methods often rely on heuristic choices to drive the optimization. In this work, we formulate SFT and alignment as a constrained optimization problem, where the LLM is trained on a task while being required to meet application-specific requirements, without resorting to heuristics. To solve this, we propose Lagrange Large Language Models (L3Ms), which employ logarithmic barriers to enforce the constraints. This approach allows for the customization of L3Ms across diverse applications while avoiding heuristic-driven processes. We demonstrate experimentally the versatility and efficacy of L3Ms in achieving tailored alignments for various applications. Large language models (LLMs) are used for a wide range of tasks: as chatbots (Brown et al., 2020; OpenAI, 2024), for code generation (Ahmad et al., 2021; Wang et al., 2021; Rozière et al., 2024), for medical assistance (Yang et al., 2022; Moor et al., 2023), and more. The key ingredients for their impressive downstream performance are supervised fine-tuning (SFT) and alignment; the former fine-tunes the LLM to a task of interest, while the latter instills it with preferential properties. Arguably, the right combination of preferential properties is highly application/task-dependent. For instance, a scholar might want a chatbot to be honest and factual to assist with their work, whereas a fiction writer might prefer the opposite behavior to help create fantastical imaginary worlds.