ChatGPT Participates in a Computer Science Exam

Indeed, there is already existing evidence to suggest that this might be the case (Bommarito and Katz, 2022; Choi et al., 2023; Kung et al., 2023; Frieder et al., 2023). However, apart from one study by legal scholars (Choi et al., 2023), existing evaluations on university exams probe the model only on a subset of the task for which it might be particularly suited (for example, excluding all questions that contain images). In addition, evaluation of the model's responses is often not blind, which can be problematic because ChatGPT is known to produce strange answers that are subject to interpretation. As such, despite much discussion about the topic, there is to this point little systematic evidence regarding the capabilities of ChatGPT on university exams (Mitchell, 2023). We present the results of a simple but rigorous experiment that evaluates the capabilities of ChatGPT on an undergraduate computer science exam about algorithms and data structures. We conducted this experiment alongside the regular university exam, which allowed us to evaluate the model's responses in a blind setup jointly with those of the students. We posed the different exam questions in a simple standardized format that allowed ChatGPT to give clear answers to all exam questions.