The Impact of Fine-tuning Large Language Models on Automated Program Repair

--Automated Program Repair (APR) uses various tools and techniques to help developers achieve functional and error-free code faster . In recent years, Large Language Models (LLMs) have gained popularity as components in APR tool chains because of their performance and flexibility. However, training such models requires a significant amount of resources. Fine-tuning techniques have been developed to adapt pre-trained LLMs to specific tasks, such as APR, and enhance their performance at far lower computational costs than training from scratch. In this study, we empirically investigate the impact of various fine-tuning techniques on the performance of LLMs used for APR. Our experiments provide insights into the performance of a selection of state-of-the-art LLMs pre-trained on code. The evaluation is done on three popular APR benchmarks (i.e., QuixBugs, Defects4J and HumanEval-Java) and considers six different LLMs with varying parameter sizes (resp. We consider three training regimens: no fine-tuning, full fine-tuning, and parameter-efficient fine-tuning (PEFT) using LoRA and IA3. We observe that full fine-tuning techniques decrease the benchmarking performance of various models due to different data distributions and overfitting. By using parameter-efficient fine-tuning methods, we restrict models in the amount of trainable parameters and achieve better results. Index T erms --large language models, automated program repair, parameter-efficient fine-tuning, AI4Code, AI4SE, ML4SE. Software development, maintenance, and evolution are expensive processes, both in terms of money and time [1]. Supporting the efficiency of software engineers responsible for these workflows can save significant resources and enable companies to speed up the production and delivery of products without loss of quality. One of the key challenges that software engineers face is the occurrence of software defects, or bugs, which are unintended errors in the code that cause deviations from expected behavior. These defects vary in complexity, from simple one-line syntax errors to intricate multi-line logic bugs that can span multiple files and components. Automated Program Repair (APR) aims to support developers with the software maintenance and evolution process, by helping them to fix any bugs they encounter and achieve their goals faster.