EPiC: Cost-effective Search-based Prompt Engineering of LLMs for Code Generation

Abstract--Large Language Models (LLMs) have seen increasing use in various software development tasks, especially in code generation. The most advanced recent methods attempt to incorporate feedback from code execution into prompts to help guide LLMs in generating correct code, in an iterative process. While effective, these methods could be costly and time-consuming due to numerous interactions with the LLM and the extensive token usage. To address this issue, we propose an alternative approach named Evolutionary Prompt Engineering for Code (EPiC), which leverages a lightweight evolutionary algorithm to evolve the original prompts toward better ones that produce high-quality code, with minimal interactions with LLM. Our evaluation against state-of-the-art (SOTA) LLM-based code generation models shows that EPiC outperforms all the baselines in terms of cost-effectiveness. Among these activities, code generation using LLMs has demonstrated significant potential. In LLM-based code generation, various prompt engineering techniques, including zero-shot [5], in-context learning [33], [34], RAG [35], and task-specific methods [36], [37], have been shown to outperform fine-tuned smaller models. The most advanced prompt engineering methods for code generation employ various agent-based approaches [28]. SOTA methods such as Reflexion [20], Language Agent Tree Search (LATS) [21], AgentCoder [22], LDB Figure 1: The initial failed prompt (left) and the mutated [23], and MetaGPT [29] are either planning-based or multicollaborative successful prompt (right) agents. While effective, these methods can be costly and time-consuming due to numerous interactions with LLMs which results in extensive token usage, making them less attractive in practical settings. The first phase involves primary code requires on average 3 minutes to generate the implementation generation using an initial prompt and its evaluation using of a function with an average of only 6 lines of code, on a set of test cases. If a correct solution is not generated, the MBPP dataset.