Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case

Traditional public opinion surveys face a number of challenges and risks related to measurement and representation dimensions, including, for example, coverage error due to incomplete frames and hard-to-reach groups, sampling error resulting from finite samples and complex designs, nonresponse error stemming from low participation and interview fatigue, measurement error introduced by questionnaire wording, and processing errors in coding and post-survey adjustments, among others (Groves, 1989; Groves and Lyberg, 2010; Weisberg, 2005). These errors could be amplified by substantial financial, human, and logistical demands, such as time spent on instrument design, piloting, and fieldwork that often forces a cost-quality trade-off that may distort population inferences. Consequently, there is a growing demand in the social sciences and market research for methods that reduce burden and cost while maintaining and improving overall data quality. Against this backdrop, Large Language Models (LLMs), trained extensively on vast and diverse data, emerge as promising alternatives for new research possibilities and applied research, including handling the abovementioned survey research limitations and measurement and representation errors. Indeed, recent advances in generative artificial intelligence (AI) suggest LLMs could serve for a number of classification tasks, including the creation of synthetic samples, providing simulated responses reflective of broader societal attitudes and behaviours (Argyle et al., 2023; Gilardi et al., 2023; González-Bustamante, 2024). The synthetic samples specifically may leverage the ability of LLMs 2 to generate contextually informed responses based on individual-level demographic characteristics and attitudes, and, in this way, potentially emulate public opinion without direct interaction with human respondents. This methodological innovation opens new avenues for rapid data collection, experimentation with sensitive topics, and a deeper understanding of complex public opinion dynamics that complement or even partially substitute for traditional surveys. Thus, the primary objective of this working paper is to evaluate the effectiveness and reliability of LLM-generated synthetic survey responses in reflecting real-world public opinion in Chile. Specifically, we aim to assess the predictive accuracy of a number of state-of-the-art private and open-source LLMs by comparing their synthetic respondents against human probabilistic responses.