Instructional Material
TrueReason: An Exemplar Personalised Learning System Integrating Reasoning with Foundational Models
Bulathwela, Sahan, Van Niekerk, Daniel, Shipton, Jarrod, Perez-Ortiz, Maria, Rosman, Benjamin, Shawe-Taylor, John
Personalised education is one of the domains that can greatly benefit from the most recent advances in Artificial Intelligence (AI) and Large Language Models (LLM). However, it is also one of the most challenging applications due to the cognitive complexity of teaching effectively while personalising the learning experience to suit independent learners. We hypothesise that one promising approach to excelling in such demanding use cases is using a \emph{society of minds}. In this chapter, we present TrueReason, an exemplar personalised learning system that integrates a multitude of specialised AI models that can mimic micro skills that are composed together by a LLM to operationalise planning and reasoning. The architecture of the initial prototype is presented while describing two micro skills that have been incorporated in the prototype. The proposed system demonstrates the first step in building sophisticated AI systems that can take up very complex cognitive tasks that are demanded by domains such as education.
The Role of Generative AI in Software Student CollaborAItion
Kiesler, Natalie, Smith, Jacqueline, Leinonen, Juho, Fox, Armando, MacNeil, Stephen, Ihantola, Petri
Collaboration is a crucial part of computing education. The increase Khan [28] has proposed an inspiring vision of how AI could in AI capabilities over the last couple of years is bound to profoundly help realize personalized individual tutors for every learner. Complementing affect all aspects of systems and software engineering, including this, an expert panel from 2020 [49] draws a scenario collaboration. In this position paper, we consider a scenario where where "AI supports orchestration of the multiple types of activities, AI agents would be able to take on any role in collaborative processes learning partners, and interaction patterns that can enrich a classroom". in computing education. We outline these roles, the activities We believe the possibilities are even broader, and to help and group dynamics that software development currently include, think about them, we propose a thought experiment that not only and discuss if and in what way AI could facilitate these roles and accommodates emerging practices and visions but also suggests activities. The goal of our work is to envision and critically examine new use cases in education that (to the best of our knowledge) have potential futures. We present scenarios suggesting how AI not yet been explored.
Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos
Hu, Kairui, Wu, Penghao, Pu, Fanyi, Xiao, Wang, Zhang, Yuanhan, Yue, Xiang, Li, Bo, Liu, Ziwei
Humans acquire knowledge through three cognitive stages: perceiving information, comprehending knowledge, and adapting knowledge to solve novel problems. Videos serve as an effective medium for this learning process, facilitating a progression through these cognitive stages. However, existing video benchmarks fail to systematically evaluate the knowledge acquisition capabilities in Large Multimodal Models (LMMs). To address this gap, we introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across six disciplines, evaluating knowledge acquisition through stage-aligned question-answer pairs: Perception, Comprehension, and Adaptation. A proposed knowledge gain metric, {\Delta}knowledge, quantifies improvement in performance after video viewing. Evaluation of LMMs reveals a steep decline in performance as cognitive demands increase and highlights a significant gap between human and model knowledge acquisition, underscoring the need for methods to enhance LMMs' capability to learn and adapt from videos.
Beyond Task Diversity: Provable Representation Transfer for Sequential Multi-Task Linear Bandits
Duong, Thang, Wang, Zhi, Zhang, Chicheng
We study lifelong learning in linear bandits, where a learner interacts with a sequence of linear bandit tasks whose parameters lie in an $m$-dimensional subspace of $\mathbb{R}^d$, thereby sharing a low-rank representation. Current literature typically assumes that the tasks are diverse, i.e., their parameters uniformly span the $m$-dimensional subspace. This assumption allows the low-rank representation to be learned before all tasks are revealed, which can be unrealistic in real-world applications. In this work, we present the first nontrivial result for sequential multi-task linear bandits without the task diversity assumption. We develop an algorithm that efficiently learns and transfers low-rank representations. When facing $N$ tasks, each played over $\tau$ rounds, our algorithm achieves a regret guarantee of $\tilde{O}\big (Nm \sqrt{\tau} + N^{\frac{2}{3}} \tau^{\frac{2}{3}} d m^{\frac13} + Nd^2 + \tau m d \big)$ under the ellipsoid action set assumption. This result can significantly improve upon the baseline of $\tilde{O} \left (Nd \sqrt{\tau}\right)$ that does not leverage the low-rank structure when the number of tasks $N$ is sufficiently large and $m \ll d$. We also demonstrate empirically on synthetic data that our algorithm outperforms baseline algorithms, which rely on the task diversity assumption.
Generation of reusable learning objects from digital medical collections: An analysis based on the MASMDOA framework
Buendรญa, Fรฉlix, Gayoso-Cabada, Joaquรญn, Sierra, Josรฉ-Luis
Learning Objects represent a widespread approach to structuring instructional materials in a large variety of educational contexts. The main aim of this work consists of analyzing from a qualitative point of view the process of generating reusable learning objects (RLOs) followed by Clavy, a tool that can be used to retrieve data from multiple medical knowledge sources and reconfigure such sources in diverse multimedia-based structures and organizations. From these organizations, Clavy is able to generate learning objects which can be adapted to various instructional healthcare scenarios with several types of user profiles and distinct learning requirements. Moreover, Clavy provides the capability of exporting these learning objects through educational standard specifications, which improves their reusability features. The analysis insights highlight the importance of having a tool able to transfer knowledge from the available digital medical collections to learning objects that can be easily accessed by medical students and healthcare practitioners through the most popular e-learning platforms.
Review for NeurIPS paper: Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization
Weaknesses: The linear combination of multi-source domain has been investigated in [a,b], which shoule be discussed. Multi-source Domain Adaptation for Face Recognition Multi-Source Domain Adaptation: A Causal View Using KL divergence or GAN to align many source domains is a quite common operation. The method seems like a simple combination of [37] and KL-divergence based alignment. Is the other divergence measure can be applied for this framework? The abstract claims that the medical image are usually heterogeneous, but the tested datasets are homogeneous.
Review for NeurIPS paper: Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization
Four knowledgeable reviewers have mixed opinions: R1 and R4 -- 'Marginally above the acceptance threshold', R3 -- 'Marginally above the acceptance threshold', and R2 -- 'Reject'. R2, who is the least confident among 4 reviewers, has a main concern about comparing with two works [a] Multi-source Domain Adaptation for Face Recognition and [b] Multi-Source Domain Adaptation: A Causal View. I read the paper and rebuttal and decide to downgrade this concern because the paper is mainly about domain generalization (DG), rather than domain adaptation (DA). There are quite differences between them and hence it is not fully necessary to compare the proposed DG approach with DA approaches thoroughly. Also, the authors agree to cite these works in future version, which to me is fine.
Ontology-Enhanced Educational Annotation Activities
Gayoso-Cabada, Joaquรญ, Goicoechea-de-Jorge, Marรญa, Gรณmez-Albarrรกn, Mercedes, Sanz-Cabrerizo, Amelia, Sarasa-Cabezuelo, Antonio, Sierra, Josรฉ-Luis
Information and communications technology and technology-enhanced learning have unquestionably transformed traditional teaching-learning processes and are positioned as key factors to promote quality education, one of the basic sustainable development goals of the 2030 agenda. Document annotation, which was traditionally carried out with pencil and paper and currently benefits from digital document annotation tools, is a representative example of this transformation. Using document annotation tools, students can enrich the documents with annotations that highlight the most relevant aspects of these documents. As the conceptual complexity of the learning domain increases, the annotation of the documents may require comprehensive domain knowledge and an expert analysis capability that students usually lack. Consequently, a proliferation of irrelevant, incorrect, and/or poorly decontextualized annotations may appear, while other relevant aspects are completely ignored by the students. The main hypothesis proposed by this paper is that the use of a guiding annotation ontology in the annotation activities is a keystone aspect to alleviate these shortcomings. Consequently, comprehension is improved, exhaustive content analysis is promoted, and meta-reflective thinking is developed. To test this hypothesis, we describe our own annotation tool, \@note, which fully implements this ontology-enhanced annotation paradigm, and we provide experimental evidence about how \@note can improve academic performance via a pilot study concerning critical literary annotation.
Knowledge Tracing in Programming Education Integrating Students' Questions
Kim, Doyoun, Kim, Suin, Jo, Yojan
Knowledge tracing (KT) in programming education presents unique challenges due to the complexity of coding tasks and the diverse methods students use to solve problems. Although students' questions often contain valuable signals about their understanding and misconceptions, traditional KT models often neglect to incorporate these questions as inputs to address these challenges. This paper introduces SQKT (Students' Question-based Knowledge Tracing), a knowledge tracing model that leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems in programming education. Our method creates semantically rich embeddings that capture not only the surface-level content of the questions but also the student's mastery level and conceptual understanding. Experimental results demonstrate SQKT's superior performance in predicting student completion across various Python programming courses of differing difficulty levels. In in-domain experiments, SQKT achieved a 33.1\% absolute improvement in AUC compared to baseline models. The model also exhibited robust generalization capabilities in cross-domain settings, effectively addressing data scarcity issues in advanced programming courses. SQKT can be used to tailor educational content to individual learning needs and design adaptive learning systems in computer science education.
Web vs. LLMs: An Empirical Study of Learning Behaviors of CS2 Students
Kumar, Aayush, Prol, Daniel, Alipour, Amin, Ragavan, Sruti Srinivasa
LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students' experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.