Learning from others' mistakes: Finetuning machine translation models with span-level error annotations

L EARNING FROM OTHERS ' MISTAKES: F INETUNING MACHINE TRANSLATION MODELS WITH SPAN-LEVEL ERROR ANNOTATIONS Lily H. Zhang 2 Hamid Dadkhahi 1 Mara Finkelstein 1 Firas Trabelsi 1 Jiaming Luo 1 Markus Freitag 1 1 Google 2 New Y ork University A BSTRACT Despite growing interest in incorporating feedback to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of utilizing fine-grained span-level annotations from offline datasets to improve model quality. We develop a simple finetuning algorithm, called Training with Annotations (TW A), to directly train machine translation models on such annotated data. TW A utilizes targeted span-level error information while also flexibly learning what to penalize within a span. Moreover, TW A considers the overall trajectory of a sequence when deciding which non-error spans to utilize as positive signals. Experiments on English-German and Chinese-English machine translation show that TW A outperforms baselines such as Supervised FineTuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data. Such data, coupled with techniques to learn from it (Christiano et al., 2017; Rafailov et al., 2023; Gulcehre et al., 2023; Dong et al., 2023), have yielded impressive results for many top language models. Most efforts, however, consider only sequence-level labels, usually in the form of a scalar score assigned to the entire output. In contrast, this work investigates the potential of using fine-grained span-level annotations from offline datasets to enhance language model training.