Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks

--Pre-trained code models are essential for various code intelligence tasks. Y et, their effectiveness is heavily influenced by the quality of the pre-training dataset, particularly human-written reference comments, which usually serve as a bridge between the programming language and natural language. One significant challenge is that such comments could become inconsistent with the corresponding code as the software evolves, leading to suboptimal model performance. Large language models (LLMs) have demonstrated superior capabilities in generating high-quality code comments. This work investigates whether substituting original human-written comments with LLM-generated ones can improve pre-training datasets for more effective pre-trained code models. As existing reference-based metrics cannot evaluate the quality of human-written reference comments themselves, to enable direct comparison between LLM-generated and human reference comments, we introduce two auxiliary tasks as novel reference-free metrics, including code-comment inconsistency detection and semantic code search. Experimental results show that LLM-generated comments exhibit superior semantic consistency with the code compared to human-written reference comments. Our manual evaluation also corroborates this conclusion, which indicates the potential of utilizing LLMs to enhance the quality of the pre-training dataset. Based on this finding, we rebuilt the CodeSearchNet dataset with LLM-generated comments and re-pre-trained the CodeT5 model. Evaluations on multiple code intelligence tasks demonstrate that models pre-trained by LLM-enhanced data outperform their counterparts (pre-trained by original human reference comments data) on code summarization, code generation, and code translation tasks. This research validates the feasibility of rebuilding the pre-training dataset by LLMs to advance code intelligence tasks. It advocates rethinking the reliance on human reference comments for code-related tasks. I NTRODUCTION In the realm of code intelligence, pre-trained code models have significantly enhanced a spectrum of tasks such as code This research was supported by the the National Key Research and Development Program of China (Grant No.2023YFB4503802), the National Natural Science Foundation of China (Grant No.62172426,62302515) and the NUDT Research Project for Student(No.ZC525Z042402). Xinjun Mao and Shangwen Wang are the corresponding authors. Pre-trained source code models such as CodeBERT [11], GraphCodeBERT [12], CodeT5 [13] and UniXcoder [14] have achieved remarkable results on various software engineering tasks and even outperform large language models when fine-tuned with domain-specific data [15]. The comments of the corresponding code snippets serve as a crucial bridge between the programming language (PL) and natural language (NL), providing contextual understanding pivots that are vital for pre-training the above models.