ChatGPTest: opportunities and cautionary tales of utilizing AI for questionnaire pretesting

Pretesting involves a small-scale trial of data collection procedures, aiming to assess them. It is a standard practice in both academic and applied research (Grimm 2010), and the output of the pretest is usually the feedback offered by interviewers on how to improve procedures and questions. The rapid advancements in generative artificial intelligence (GAI) have opened up new avenues for enhancing various aspects of research, including the design and evaluation of survey questionnaires. AI technologies like large language models (LLMs) have demonstrated remarkable potential in generating human-like text, offering a promising approach to pretesting survey instruments. This article explores the use of GPT models as a tool for pretesting survey questionnaires. Illustrated with two applications, it suggests incorporating GPT feedback as an additional stage before human pretesting, potentially reducing successive iterations. However, the article emphasizes the indispensable role of researchers' judgment in implementing AIgenerated feedback. GPT is an LLM that utilizes advanced algorithms to generate texts that mimic the syntax, semantics, and grammar of human writing, which are approximated by statistical patterns learned from training data (for a technical review, see OpenAI 2023). Like most of the LLMs, GPT models predict the next word in a sequence based on the preceding words.