Concerns on Bias in Large Language Models when Creating Synthetic Personae

One immense concern relates to the existence of bias in the models, and creating synthetic personae has the potential to aid the investigation of how different forms of bias manifest in LLMs, by introducing a new method of testing. However, the black-box nature of a majority of these models, and their inability to express'opinions' contrary to overall LLM rules or fail-safes, introduces complexities in how to prompt the models to act out specific synthetic personae in various scenarios. This position paper introduces an exploration of a few fundamental questions: What are the benefits and drawbacks of using synthetic personae in HCI research, and how can we customize them beyond the limitations of current LLMs? The perspectives presented in this paper have sprung from the sub-study of a PhD project on Artificial Intelligence and Participatory Design [18]. The sub-study, currently a work in progress, aims at developing a novel method of adversarial testing [6, 13, 21] through the use of contextualized"real-life" vignettes [2, 16] prompted to the interfaces of multiple LLMs to identify potential bias, trying to open up the"black box" from a more qualitative human-computer interaction perspective[10]. 2 BIAS DETECTION IN LLM INTERFACES Research in various sub-fields has shown that human engagement in AI design, development, and evaluation, particularly in a qualitative manner, can ensure a focus on the socio-technical embeddedness of AI [3].