Prompting Considered Harmful

Prompting is not the same as natural language. When people converse with each other, they work together to communicate, forming mental models of a conversation partner's communicative intent based not only on words but also on paralinguistic and other contextual cues, theory-of-mind abilities, and by requesting clarification as needed.4 By contrast, while some prompts resemble natural language, many of the most "successful" prompts do not--for instance, image generation is a domain where arcane prompts tend to produce better results than those in plain language.1 Further, prompts are surprisingly sensitive to variations in wording, spelling, and punctuation in ways that lead to substantial changes in model outputs, whereas these same permutations would be unlikely to impact human interpretation of intent--for example, jailbreak prompts using suffix attacks10 or word-repetition commands.7 While some prompts resemble natural language, many of the most "successful" prompts do not.