Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments

The advent of Large Language Models (LLMs), such as OpenAI's ChatGPT, Codex, and GitHub's Copilot, affects the educational landscape at its core, as LLMs offer entirely new possibilities, but also challenges for educators, learners, and institutions. Even though LLMs have only appeared very recently to a broader audience, research has started to address their implications on computing education, particularly programming. The generative potential may be used by educators for the design of new programming tasks [Sa22], or for students to gather formative feedback [Ka23, Zh22]. At the same time, implications for programming pedagogy and assessments are being discussed [Be23, BK23, RTT23], as the lowthreshold availability of LLMs raises new questions with regard to adequate task designs, students' contribution, plagiarism, and ethical conduct. Educators and institutions will soon need to reconsider the design of (formative) assessments. In this context, it is crucial to investigate the capabilities and limitations of LLMs for novice learners of programming, whose challenges have a well-documented history [SS86, Mc01, Lu18].