Using machine learning to measure evidence of students' sensemaking in physics courses

Teaching and instruction in undergraduate physics courses has largely relied on problem-solving as the standard method to measure student performance [1-6]. Common practice is for "real-time" performance to be measured via multiple-choice or single-solution problems, where canonically correct answers determine the student's knowledge of the core material. Accuracy scores across assignments and examinations, typically coupled with letter grades, act as signals of progress throughout the course as well as final verdicts of student success. While engaging in problem-solving is a useful experience for students in a physics classroom, using the problem solution as a measure of student learning assumes a direct correlation that may not always hold. Problem-solving accuracy as a measurand assumes that students will engage in a learning process involving the core material to obtain the problem solution. Often times, there are alternative strategies for obtaining a problem solution such as rote-memorization of the rules or procedures required for solving similar problem types [7]. In this scenario, students would score very high on exams that contain these problem types; however given a previously unseen problem structure where the same core material is to be applied, the students would struggle. Here, a risk of using problem-solving accuracy as the predominant metric is an inflated sense of confidence in both the instructor and the student that the core material has been learned. It could also pose a risk for confounding variables in research studies that aim to investigate how instructional techniques influence student learning [8-12].