On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts

The amount of scholarly texts is consistently increasing; around 2.5 million research articles are published yearly (Rabby et al., 2024). Due to this enormous increase, the classification of (scientific) texts has been attracting even more attention in recent years (Born-mann et al., 2021). Classifying the research area of scientific texts requires significant domain knowledge in various complex research fields. Hence, manual classification is challenging and time-consuming for librarians and limits the number of texts that can be classified manually (Zhang et al., 2023). Moreover, due to complex hierarchical classification schemes and their existing variety, classification of publications is also an unbeloved activity for researchers. Prominent examples of classification schemes include the Open Research Knowledge Graph (ORKG) (Auer and Mann, 2019), Microsoft Academic Graph (Wang et al., 2020), the Semantic Scholar Academic Graph (Kinney et al., 2023), ACM computing classification system (Rous, 2012), Dewey Decimal Classification (DDC) (Scott, 1998), and the ACL Anthology (Bird et al., 2008).