A Case Study in Engineering a Conversational Programming Assistant's Persona

One particularly interesting aspect of these models is that their behavior can be configured by a prompt, the initial text provided to the model, which establishes a pattern that the model attempts to continue. General purpose Large Language models can be fine-tuned on specific corpora to provide expertise in a particular domain. One such model is the OpenAI Codex model [3], a 12 billion parameter version of GPT-3 [2, 11], fine-tuned on code samples from 54 million public software repositories on GitHub. This model powers Github Co-Pilot [5], which primarily provides code-completion services within an Integrated Development Environment. We wondered whether such a model could power a conversational programming assistant and perhaps approach the vision laid out by Rich and Waters for their Programmer's Apprentice [15]. We developed the Programmer's Assistant prototype to explore this possibility, and to test whether potential users would find this sort of system useful and desirable [16]. In this paper we will review the steps taken to engineer the prompt for the Programmer's Assistant that used the Codex model to power an interactive conversational assistant, and how we evolved the prompt to establish the desired persona and behavior.