PHOENIX: Open-Source Language Adaption for Direct Preference Optimization

Large Language Models like GPT-3 [1], Llama-2 [2] or Mistral [3] demonstrate very good results in solving a wide range of tasks and form good foundation models for adaptation to chat models [4], [5] or other domain-specific applications. These and other open-source models benefit from a wide range of technological improvements. Structural improvements such as Grouped-Query Attention [6] or Flash-Attention [7], [8] for example were able to drastically reduce both the calculation time and the memory overhead of the attention block. Furthermore, high-quality closed-source models such as GPT-4[9], ChatGPT [10], Claude 2 [11] and Crowdsourcing efforts [12] allowed the creation of high-quality datasets. These models act as teachers, distilling the information and allowing the creation of student models trained with this distilled information. However, despite this highly diverse field of research, which, in addition to the points already mentioned, also deals with inference optimization i.a.