Automated Text Scoring in the Age of Generative AI for the GPU-poor

Generative language models (GLMs), such as GPT-4 [34] and Claude [2], have demonstrated powerful performance across a variety of language and reasoning tasks. In the field of education, researchers are exploring the extent to which these models can perform tasks such as automated essay scoring [56], providing feedback to students [4], individual tutoring [7], and more [15]. Although GLMs show promise in automating certain educative tasks, there are critical limitations that hinder the possibility of wider implementation. For instance, researchers have shown that GLMs can be "jail-broken" to bypass safety guardrails [58] and can disclose personally identifiable information. Large GLMs are extremely large, requiring millions of dollars to train and deploy; as such, they are highly inefficient for specialized tasks [26]. These models are constantly being updated, sometimes leading to degraded performance [6], and they are only accessible via Application Programming Interfaces (APIs), which lead to issues around replicability and leave little room to conduct rigorous research. It is for these reasons that we shift the focus away from large, proprietary GLMs toward smaller, open-source GLMs. In this study, we focus on two educational applications: Automated Text Scoring (ATS) and providing feedback--specifically, feedback that justifies scores based on the scoring rubric. Our study is the first to demonstrate that it is possible to efficiently fine-tune such GLMs to yield high-quality scores, and that (at least some) feedback from fine-tuned models can explain these scores.