From Prompt Engineering to Prompt Science with Humans in the Loop

In recent years, as the sophistication and capabilities of large language models (LLMs) have grown, so have the tasks for which they're applicable, going beyond information extraction and synthesis15 to include analysis, content creation, and reasoning.8 Unsurprisingly, many researchers find them useful for research tasks, such as identifying relevant papers,19 synthesizing literature reviews,3 writing proposals,11 and analyzing data.31 They have also been found effective for investigative tasks, such as drug discovery.35 There is growing concern, however, that a large portion of this success hinges on prompt engineering, which is often an ad-hoc method to revise prompts being fed into an LLM to achieve a desired response or analysis.24 LLMs are increasingly being used in scientific research, but their application often involves ad-hoc decisions that can impact research quality.