CodeShell Technical Report

Code Large Language Models (CodeLLM) such as CodeGen (Nijkamp et al., 2023), CodeLlama (Rozière et al., 2023), and StarCoder (Li et al., 2023) have revolutionized software development by automating tasks, reducing errors, and improving efficiency (OpenAI, 2023). Leveraging deep learning and vast code datasets (Nijkamp et al., 2023; Kocetkov et al., 2022), these models enhance developer productivity and make software development more accessible to a broader audience. Current CodeLLMs can broadly be divided into three main categories: pre-training from scratch (Li et al., 2023), pre-training from an existing LLM (Chen et al., 2021; Rozière et al., 2023), and Instruct Tuning (Luo et al., 2023). Models that are pre-trained from scratch require a substantial volume of data and considerable time (Li et al., 2023; Touvron et al., 2023). On the other hand, pre-training from an existing LLM leverages a pre-existing model as its foundation, which brings the benefits of shortened training times and improved efficiency with less data (Chen et al., 2021; Rozière et al., 2023).