Instructional Material
Ontology-Enhanced Educational Annotation Activities
Gayoso-Cabada, Joaquí, Goicoechea-de-Jorge, María, Gómez-Albarrán, Mercedes, Sanz-Cabrerizo, Amelia, Sarasa-Cabezuelo, Antonio, Sierra, José-Luis
Information and communications technology and technology-enhanced learning have unquestionably transformed traditional teaching-learning processes and are positioned as key factors to promote quality education, one of the basic sustainable development goals of the 2030 agenda. Document annotation, which was traditionally carried out with pencil and paper and currently benefits from digital document annotation tools, is a representative example of this transformation. Using document annotation tools, students can enrich the documents with annotations that highlight the most relevant aspects of these documents. As the conceptual complexity of the learning domain increases, the annotation of the documents may require comprehensive domain knowledge and an expert analysis capability that students usually lack. Consequently, a proliferation of irrelevant, incorrect, and/or poorly decontextualized annotations may appear, while other relevant aspects are completely ignored by the students. The main hypothesis proposed by this paper is that the use of a guiding annotation ontology in the annotation activities is a keystone aspect to alleviate these shortcomings. Consequently, comprehension is improved, exhaustive content analysis is promoted, and meta-reflective thinking is developed. To test this hypothesis, we describe our own annotation tool, \@note, which fully implements this ontology-enhanced annotation paradigm, and we provide experimental evidence about how \@note can improve academic performance via a pilot study concerning critical literary annotation.
Knowledge Tracing in Programming Education Integrating Students' Questions
Kim, Doyoun, Kim, Suin, Jo, Yojan
Knowledge tracing (KT) in programming education presents unique challenges due to the complexity of coding tasks and the diverse methods students use to solve problems. Although students' questions often contain valuable signals about their understanding and misconceptions, traditional KT models often neglect to incorporate these questions as inputs to address these challenges. This paper introduces SQKT (Students' Question-based Knowledge Tracing), a knowledge tracing model that leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems in programming education. Our method creates semantically rich embeddings that capture not only the surface-level content of the questions but also the student's mastery level and conceptual understanding. Experimental results demonstrate SQKT's superior performance in predicting student completion across various Python programming courses of differing difficulty levels. In in-domain experiments, SQKT achieved a 33.1\% absolute improvement in AUC compared to baseline models. The model also exhibited robust generalization capabilities in cross-domain settings, effectively addressing data scarcity issues in advanced programming courses. SQKT can be used to tailor educational content to individual learning needs and design adaptive learning systems in computer science education.
Web vs. LLMs: An Empirical Study of Learning Behaviors of CS2 Students
Kumar, Aayush, Prol, Daniel, Alipour, Amin, Ragavan, Sruti Srinivasa
LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students' experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.
Reviews: Online Continual Learning with Maximal Interfered Retrieval
This paper describes an approach to improve rehearsal-based continual learning techniques (either replay-based or with a generative model) by identifying samples that are most useful to avoid forgetting. This is achieved by computing the increase in loss on the replayed samples, and using this to determine which samples should be used during learning. It is a simple and intuitive idea, the paper is clearly written, and experiments on multiple datasets are compelling. I think it could make a nice addition to the conference, but needs a few improvements first. My main criticism is that the approach requires a separate virtual gradient step for each actual step, to compute the change in loss on the replay samples.
Review for NeurIPS paper: Neural Networks Fail to Learn Periodic Functions and How to Fix It
Weaknesses: A significant shortcoming in the approach is the lack of proper and thorough validation with recurrent neural networks. The stated problem (i.i.d. in fourier space, restricted to a compact region in R d) can be tackled with the auto-regressive approach provided by RNNs. However this is not mentioned, or compared against in the paper. I note that RNNs are somewhat compared against in the appendix, but this is with respect to RNN Snake vs RNN. Above all this paper needs a comparison showing the ability of feedforward Snake networks to outperform vanilla RNNs.
Review for NeurIPS paper: Neural Networks Fail to Learn Periodic Functions and How to Fix It
I think this is an interesting submission, that lead to a detailed discussion among the reviewers. Overall the work is novel and looks at an interesting question regarding extrapolation (and dealing with periodic functions). Overall I agree with some of the reviewers that the motivation and generally the write-up of the work could be improved, but I think there is already value in the work. I would like to highlight a few points that are worth considering: * a further discussion regarding how the proposed approach compares to RNNs or autoregressive models when it comes to modeling periodicity * there are some concerns regarding the methodology used (e.g.
Review for NeurIPS paper: Improved Schemes for Episodic Memory-based Lifelong Learning
There has been a plethora of recent and historical work on this topic, finding different ways to help networks alleviate the issue of catastrophic forgetting --- where a network trained on tasks A_0 through A_i, forgets these to differing degrees when trained on tasks A_i 1 onward. Most methods can be divided into regularisation based, memory based or meta-learning based. One relatively recent work is GEM (gradient of episodic memory) (and relatedly A-GEM). This works by storing examples from seen tasks in an episodic memory. When learning a new task, the gradient update is modified such that it does not increase the loss on examples from previous tasks (these are represented by the examples in memory).
Review for NeurIPS paper: Improved Schemes for Episodic Memory-based Lifelong Learning
The paper introduces a clear, simple generalisation of two established continual learning methods (GEM and A-GEM) which performs very well in a thorough empirical evaluation. All reviewers and the AC value the effort that the authors put in their response. There is consensus that the work has merit and all reviewers recommend accepting the paper (R1 and R4 raised their score).
LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations
Abu-Rasheed, Hasan, Jumbo, Constance, Amin, Rashed Al, Weber, Christian, Wiese, Veit, Obermaisser, Roman, Fathi, Madjid
While learning personalization offers great potential for learners, modern practices in higher education require a deeper consideration of domain models and learning contexts, to develop effective personalization algorithms. This paper introduces an innovative approach to higher education curriculum modelling that utilizes large language models (LLMs) for knowledge graph (KG) completion, with the goal of creating personalized learning-path recommendations. Our research focuses on modelling university subjects and linking their topics to corresponding domain models, enabling the integration of learning modules from different faculties and institutions in the student's learning path. Central to our approach is a collaborative process, where LLMs assist human experts in extracting high-quality, fine-grained topics from lecture materials. We develop a domain, curriculum, and user models for university modules and stakeholders. We implement this model to create the KG from two study modules: Embedded Systems and Development of Embedded Systems Using FPGA. The resulting KG structures the curriculum and links it to the domain models. We evaluate our approach through qualitative expert feedback and quantitative graph quality metrics. Domain experts validated the relevance and accuracy of the model, while the graph quality metrics measured the structural properties of our KG. Our results show that the LLM-assisted graph completion approach enhances the ability to connect related courses across disciplines to personalize the learning experience. Expert feedback also showed high acceptance of the proposed collaborative approach for concept extraction and classification.
CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification
Patrício, Cristiano, Rio-Torto, Isabel, Cardoso, Jaime S., Teixeira, Luís F., Neves, João C.
The main challenges limiting the adoption of deep learning-based solutions in medical workflows are the availability of annotated data and the lack of interpretability of such systems. Concept Bottleneck Models (CBMs) tackle the latter by constraining the final disease prediction on a set of predefined and human-interpretable concepts. However, the increased interpretability achieved through these concept-based explanations implies a higher annotation burden. Moreover, if a new concept needs to be added, the whole system needs to be retrained. Inspired by the remarkable performance shown by Large Vision-Language Models (LVLMs) in few-shot settings, we propose a simple, yet effective, methodology, CBVLM, which tackles both of the aforementioned challenges. First, for each concept, we prompt the LVLM to answer if the concept is present in the input image. Then, we ask the LVLM to classify the image based on the previous concept predictions. Moreover, in both stages, we incorporate a retrieval module responsible for selecting the best examples for in-context learning. By grounding the final diagnosis on the predicted concepts, we ensure explainability, and by leveraging the few-shot capabilities of LVLMs, we drastically lower the annotation cost. We validate our approach with extensive experiments across four medical datasets and twelve LVLMs (both generic and medical) and show that CBVLM consistently outperforms CBMs and task-specific supervised methods without requiring any training and using just a few annotated examples. More information on our project page: https://cristianopatricio.github.io/CBVLM/.