Experiments of pretraining 410M and 1B models on the C4 dataset demonstrate that MA TES significantly outperforms random data selection on extensive downstream tasks. It doubles the gains achieved by the state-of-the-art data selection approach that leverages larger reference models and reduces the total FLOPs required to reach certain performances by half. Further analyses validate the effectiveness of the locally probed oracle data influence and the approximation with data influence models. Our code is open-sourced at https://github.com/cxcscmu/MA

The pre-trained Large Language Models (LLMs) can be adapted for many downstream tasks and tailored to align with human preferences through fine-tuning.

Alpaca, created by applying SelectIT to the Alpaca-GPT4 dataset.

SwapPrompt employs a dual prompts paradigm, i.e., an online prompt and a target prompt

Among all the contributing factors, the quality and selection of data is becoming increasingly recognized for its importance in training LLMs effectively.

Selecting a suitable pretraining dataset is crucial for both general-domain (e.g., GPT -

We revisit data selection in a modern context of finetuning from a fundamental perspective.