Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models

The advancement of Large Language Models (LLMs) has opened up new avenues for tackling annotation tasks in the field of social sciences. These models, especially the newer iterations like Chat-GPT or GPT-4, are now being used to annotate textual data (Gilardi, Alizadeh, & Kubli, 2023; Heseltine & Hohenberg, 2023; Møller, Dalsgaard, Pera, & Aiello, 2023; Ziems et al., 2023), which can be helpful for analyzing various social and political phenomena (Törnberg, 2023; Ziems et al., 2023). However, a significant challenge arises when there is a necessity to share research data with proprietary and closed models that are provided by companies due to the utilization of APIs (Ollion, Shen, Macanovic, & Chatelain, 2023; Spirling, 2023). This is particularly concerning in scenarios where data sharing is not preferable due to data privacy. In light of this, open models which can be operated on independent devices like university servers, present a viable alternative (Alizadeh et al., 2023). They allow researchers to harness the potential of generative large language models without compromising data security. This paper endeavors to promote the adoption of open models by providing two examples and guidelines for leveraging them instead of proprietary models for annotation tasks within the social sciences.