PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation

With the right design [46], such interfaces could enable experts to steer the output of LLMs toward content that better aligns with the nuances and needs of their domains, and transform the role of the subject matter expert from a producer to a curator--a competent and critical judge who instructs the AI agent on what is needed, evaluates the output, and iterates on the instructions until the results are satisfactory. Instead of replacing human experts, these interfaces could help bridge human intelligence with machine intelligence to dramatically reduce the time and effort required to create content that adheres to expert tastes and standards. To realize the producer-to-curator shift and integrate domain expertise more closely into prompt engineering, we need authoring interfaces that: (i) deeply embed LLMs within existing expert workflows, augmenting content creation with carefully scaffolded interface support for prompt engineering; (ii) encourage experimentation on many prompt variations to systematically test the impact of changes in instructional wording on model output; (iii) offer mechanisms for curating prompt formulations that work well at various levels of abstraction; (iv) integrate generation into the publishing workflow. However, designing authoring interfaces that support experts across all four fronts is difficult as LLMs pose unique usability challenges tied to high metacognitive demands during prompt construction [45], and users can struggle to get the models to integrate well with their existing workflow as even small perturbations such as adding a space at the end of a prompt can cause the LLM to change its output [37]. For domain experts who aren't AI specialists, recent literature on prompt engineering has also highlighted how designing effective prompts can be surprisingly difficult for non-AI experts [8, 51].