A Survey on Pretrained Language Models for Neural Code Intelligence

Transformer models on modeling sequential Programming languages (Pierce, 2002) serve as data (Krizhevsky et al., 2017; Vaswani et al., the foundation of software, enabling humans to 2017). Motivated by the software naturalness hypothesis communicate with computers and instruct them to (Hindle et al., 2016; Buratti et al., 2020), perform computation. The process of developing which suggests that programming languages can be software using programming languages, known as understood and generated like natural languages, software development, has become a thriving industry researchers have treated source code as sequential that plays a crucial role in the modern digital data and applied sequential neural architectures, world. However, software development involves a like the Transformer model (Vaswani et al., range of tasks beyond programming, including testing, 2017), to understand and generate programs (Feng documentation writing, and bug fixing, which et al., 2020; Guo et al., 2021). In the Natural Language are known to be challenging and require a high Processing (NLP) community, it has been level of human expertise (Brooks, 1978).