DELIFT: Data Efficient Language model Instruction Fine Tuning

Fine-tuning large language models (LLMs) is essential for enhancing their performance on specific tasks but is often resource-intensive due to redundant or uninformative data. To address this inefficiency, we introduce DELIFT (Data Efficient Language model Instruction Fine-Tuning), a novel algorithm that systematically optimizes data selection across the three key stages of fine-tuning: (1) instruction tuning, (2) task-specific fine-tuning (e.g., reasoning, question-answering), and (3) continual fine-tuning (e.g., incorporating new data versions). Unlike existing methods that focus on single-stage optimization or rely on computationally intensive gradient calculations, DELIFT operates efficiently across all stages. Central to our approach is a pairwise utility metric that quantifies how beneficial a data sample is for improving the model's responses to other samples, effectively measuring the informational value relative to the model's current capabilities. By leveraging different submodular functions applied to this metric, DELIFT selects diverse and optimal subsets that are useful across all stages of fine-tuning. Experiments across various tasks and model scales demonstrate that DELIFT can reduce the fine-tuning data size by up to 70% without compromising performance, offering significant computational savings and outperforming existing methods in both efficiency and efficacy. Fine-tuning large language models (LLMs) is pivotal for adapting these powerful architectures (Devlin et al., 2019; Brown et al., 2020a; Touvron et al., 2023) to specialized tasks such as intricate reasoning, precise question-answering, and the seamless integration of new information (Ouyang et al., 2022). This transformation--from a general-purpose model to a task-specific agent--heavily relies on the quality and nature of the data employed during fine-tuning, which critically determines the model's subsequent performance (Wei et al., 2022; Zhou et al., 2023; Hoffmann et al., 2024). The effectiveness of fine-tuning hinges on the quality, diversity, and relevance of the selected data (Gururangan et al., 2020; Wei et al., 2022; Zhou et al., 2023). High-quality data ensures accurate learning, diverse data enhances generalization, and relevant data aligns the model's capabilities with specific application needs. However, optimizing data selection across different fine-tuning phases remains a significant challenge, leading to our central research question: How can we create a unified framework for efficient data selection across all fine-tuning stages of LLMs, while optimizing performance and maximizing data efficiency? To address this challenge, we present DELIFT (Data Efficient Language model Instruction Fine-Tuning), a novel, unified, and computationally efficient algorithm engineered to optimize data selection across all stages of the fine-tuning process.