The threat of analytic flexibility in using large language models to simulate human data: A call to attention

Social scientists are now using large language models to create " silicon samples" - synthetic datasets intended to stand in for human respondents, aimed at revolutionising human subjects research. However, there are many analytic choices which must be made to produce these samples. Though many of these choices are defen sible, their impact on sample quality is poorly understood . I map out these analytic choices and demonstrate how a very small number of decisions can dramatically change the correspondence between silicon samples and human data. Configurations (N = 252) varied substantially in their capacity to estimate (i) rank ordering of participants, (ii) response distributions, and (iii) between - scale correlations. Most critically, configurations were not consistent in quality: those that performed well on one dimension often performed poorly on another, implying that there i s no "one - size - fits - all" configuration that optimises the accuracy of these samples. I call for greater attention to the threat of analytic flexibility in using silicon samples. Generative large language models (LLMs) have ushered in a sea change within the social sciences. The capacity of these models to produce sensical natural language output to queries, as well as their flexibility, efficiency, and generalisability, has led re searchers to explore their use in tasks across the research process. One of the most ambitious proposed uses of LLMs in this respect has been to use them as simulated research participants to determine how human participants might respond to a particular context, set of survey questions, or intervention (Cui et al., 2025). This is based on the tacit assumption that, on some level, the corpuses of data which LLMs are trained on may enable them to approximate similar data - generating processes to those which underpin human responses (Z. If achievable, this could have a transformative impact on the social sciences.