Large Language Models for Code Summarization

The introduction of Encoder-Decoder architectures in natural language processing [26] (both recurrent [6] and Transformer-based [29]) has motivated researchers to apply them to software engineering. One important application is generating summaries of code [25, 2, 11]. A code summarization tool is useful for example to understand legacy code or to create documentation. Since the spread of Large Language Models (LLMs), the working programmer has many more opportunities to use deep learning-based tools. Closed models (such as GPT-4 [21] or Gemini [27]) and open models (such as CodeLlama [24] or WizardCoder [19]) demonstrate impressive capabilities of generating source code based on a task description, as well as generating natural-language summary of code. The main objective of this technical report is to investigate how well open-sourced LLMs handle source code in relation with natural language text. In particular, we discuss results of some of the most acknowledged open-source LLMs, focusing on their code summarization/explanation (code-to-text) capabilities. We also discuss code generation (text-to-code) capabilities of these LLMs, as this is often considered to be their most defining capability. That is, LLMs are often ranked simply based on results on a code generation benchmark.