Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Ivison, Hamish, Smith, Noah A., Hajishirzi, Hannaneh, Dasigi, Pradeep
–arXiv.org Artificial Intelligence
Obtaining labeled data to train a model for a task of interest is often expensive. Prior work shows training models on multitask data augmented with task descriptions (prompts) effectively transfers knowledge to new tasks. Towards efficiently building task-specific models, we assume access to a small number (32-1000) of unlabeled target-task examples and use those to retrieve the most similar labeled examples from a large pool of multitask data augmented with prompts. Compared to the current practice of finetuning models on uniformly sampled prompted multitask data (e.g.: FLAN, T0), our approach of finetuning on cross-task nearest neighbors is significantly more data-efficient. Using only 2% of the data from the P3 pool without any labeled target-task data, our models outperform strong baselines trained on all available data by 3-30% on 12 out of 14 datasets representing held-out tasks including legal and scientific document QA. Similarly, models trained on cross-task nearest neighbors from SuperNaturalInstructions, representing about 5% of the pool, obtain comparable performance to state-of-the-art models on 12 held-out tasks from that pool. Moreover, the models produced by our approach also provide a better initialization than single multitask finetuned models for few-shot finetuning on target-task data, as shown by a 2-23% relative improvement over few-shot finetuned T0-3B models on 8 datasets.
arXiv.org Artificial Intelligence
May-24-2023
- Country:
- Africa > Middle East
- Egypt (0.14)
- Asia > Middle East
- Palestine (0.14)
- Europe > United Kingdom
- Northern Ireland (0.14)
- North America > United States
- Minnesota (0.14)
- New York > New York County
- New York City (0.14)
- Africa > Middle East
- Genre:
- Research Report
- New Finding (0.46)
- Promising Solution (0.34)
- Research Report
- Industry:
- Government (1.00)
- Law (1.00)
- Leisure & Entertainment > Sports
- Football (1.00)
- Media > Television (0.67)
- Technology: