Instructional Material
Learning-by-teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education
Chen, Angxuan, Wei, Yuang, Le, Huixiao, Zhang, Yan
This study investigates the potential of using ChatGPT as a teachable agent to support students' learning by teaching process, specifically in programming education. While learning by teaching is an effective pedagogical strategy for promoting active learning, traditional teachable agents have limitations, particularly in facilitating natural language dialogue. Our research explored whether ChatGPT, with its ability to engage learners in natural conversations, can support this process. The findings reveal that interacting with ChatGPT improves students' knowledge gains and programming abilities, particularly in writing readable and logically sound code. However, it had limited impact on developing learners' error-correction skills, likely because ChatGPT tends to generate correct code, reducing opportunities for students to practice debugging. Additionally, students' self-regulated learning (SRL) abilities improved, suggesting that teaching ChatGPT fosters learners' higher self-efficacy and better implementation of SRL strategies. This study discussed the role of natural dialogue in fostering socialized learning by teaching, and explored ChatGPT's specific contributions in supporting students' SRL through the learning by teaching process. Overall, the study highlights ChatGPT's potential as a teachable agent, offering insights for future research on ChatGPT-supported education.
Educational-Psychological Dialogue Robot Based on Multi-Agent Collaboration
Intelligent dialogue systems are increasingly used in modern education and psychological counseling fields, but most existing systems are limited to a single domain, cannot deal with both educational and psychological issues, and often lack accuracy and professionalism when dealing with complex issues. To address these problems, this paper proposes an intelligent dialog system that combines educational and psychological counseling functions. The system consists of multiple AI agent, including security detection agent, intent identification agent, educational LLM agent, and psychological LLM agent, which work in concert to ensure the provision of accurate educational knowledge Q\&A and psychological support services. Specifically, the system recognizes user-input intentions through an intention classification model and invokes a retrieval-enhanced educational grand model and a psychological grand model fine-tuned with psychological data in order to provide professional educational advice and psychological support.
Good practices for evaluation of machine learning systems
Ferrer, Luciana, Scharenborg, Odette, Bäckström, Tom
Many development decisions affect the results obtained from ML experiments: training data, features, model architecture, hyperparameters, test data, etc. Among these aspects, arguably the most important design decisions are those that involve the evaluation procedure. This procedure is what determines whether the conclusions drawn from the experiments will or will not generalize to unseen data and whether they will be relevant to the application of interest. If the data is incorrectly selected, the wrong metric is chosen for evaluation or the significance of the comparisons between models is overestimated, conclusions may be misleading or result in suboptimal development decisions. To avoid such problems, the evaluation protocol should be very carefully designed before experimentation starts. In this work we discuss the main aspects involved in the design of the evaluation protocol: data selection, metric selection, and statistical significance. This document is not meant to be an exhaustive tutorial on each of these aspects. Instead, the goal is to explain the main guidelines that should be followed in each case. We include examples taken from the speech processing field, and provide a list of common mistakes related to each aspect.
How Good is ChatGPT in Giving Adaptive Guidance Using Knowledge Graphs in E-Learning Environments?
Ocheja, Patrick, Flanagan, Brendan, Dai, Yiling, Ogata, Hiroaki
E-learning environments are increasingly harnessing large language models (LLMs) like GPT-3.5 and GPT-4 for tailored educational support. This study introduces an approach that integrates dynamic knowledge graphs with LLMs to offer nuanced student assistance. By evaluating past and ongoing student interactions, the system identifies and appends the most salient learning context to prompts directed at the LLM. Central to this method is the knowledge graph's role in assessing a student's comprehension of topic prerequisites. Depending on the categorized understanding (good, average, or poor), the LLM adjusts its guidance, offering advanced assistance, foundational reviews, or in-depth prerequisite explanations, respectively. Preliminary findings suggest students could benefit from this tiered support, achieving enhanced comprehension and improved task outcomes. However, several issues related to potential errors arising from LLMs were identified, which can potentially mislead students. This highlights the need for human intervention to mitigate these risks. This research aims to advance AI-driven personalized learning while acknowledging the limitations and potential pitfalls, thus guiding future research in technology and data-driven education.
Selective Reviews of Bandit Problems in AI via a Statistical View
Zhou, Pengjie, Wei, Haoyu, Zhang, Huiming
Introduction Reinforcement Learning (RL) is one of the most prominent and widely discussed methods in artificial intelligence, primarily focusing on how an agent learns to make decisions by interacting with an environment to maximize cumulative rewards [1]. RL has seen extensive applications in various domains, including autonomous driving [2], recommendation systems [3], unmanned aerial vehicles (UAVs) [4], financial trading [5], causal inference [6], and precision medicine [7,8]; see [9,10] for a review. The classic and simplified problem in RL is the stochastic bandit problems. Stochastic bandit problems exemplify the exploration-exploitation tradeoff dilemma, where an agent must choose between exploring new options to gather more information and exploiting known options to maximize rewards. The current review literature on stochastic bandit algorithms highlights applications in areas such as recommendation systems[11-13], experimental design[14], and precision medicine[8], causal inference[15]. Efficient bandit algorithms are designed from a statistical perspective. However, these aspects remain underexplored in existing reviews. This paper aims to address this gap by focusing on the probabilistic and statistical foundations of stochastic algorithms, with particular emphasis on concentration inequalities, minimax rate of regret upper bounds, small-sample statistical inferences, linear models, Bayesian optimization, statistical learning theory, design of experiments, the Neyman-Rubin causal model, functional data analysis, robust statistics, information theory, and so on.
A Study on Quantum Neural Networks in Healthcare 5.0
The working environment in healthcare analytics is transforming with the emergence of healthcare 5.0 and the advancements in quantum neural networks. In addition to analyzing a comprehensive set of case studies, we also review relevant literature from the fields of quantum computing applications and smart healthcare analytics, focusing on the implications of quantum deep neural networks. This study aims to shed light on the existing research gaps regarding the implications of quantum neural networks in healthcare analytics. We argue that the healthcare industry is currently transitioning from automation towards genuine collaboration with quantum networks, which presents new avenues for research and exploration. Specifically, this study focuses on evaluating the performance of Healthcare 5.0, which involves the integration of diverse quantum machine learning and quantum neural network systems. This study also explores a range of potential challenges and future directions for Healthcare 5.0, particularly focusing on the integration of quantum neural networks.
Scaffold or Crutch? Examining College Students' Use and Views of Generative AI Tools for STEM Education
Wang, Karen D., Wu, Zhangyang, Tufts, L'Nard II, Wieman, Carl, Salehi, Shima, Haber, Nick
Developing problem-solving competency is central to Science, Technology, Engineering, and Mathematics (STEM) education, yet translating this priority into effective approaches to problem-solving instruction and assessment remain a significant challenge. The recent proliferation of generative artificial intelligence (genAI) tools like ChatGPT in higher education introduces new considerations about how these tools can help or hinder students' development of STEM problem-solving competency. Our research examines these considerations by studying how and why college students use genAI tools in their STEM coursework, focusing on their problem-solving support. We surveyed 40 STEM college students from diverse U.S. institutions and 28 STEM faculty to understand instructor perspectives on effective genAI tool use and guidance in STEM courses. Our findings reveal high adoption rates and diverse applications of genAI tools among STEM students. The most common use cases include finding explanations, exploring related topics, summarizing readings, and helping with problem-set questions. The primary motivation for using genAI tools was to save time. Moreover, over half of student participants reported simply inputting problems for AI to generate solutions, potentially bypassing their own problem-solving processes. These findings indicate that despite high adoption rates, students' current approaches to utilizing genAI tools often fall short in enhancing their own STEM problem-solving competencies. The study also explored students' and STEM instructors' perceptions of the benefits and risks associated with using genAI tools in STEM education. Our findings provide insights into how to guide students on appropriate genAI use in STEM courses and how to design genAI-based tools to foster students' problem-solving competency.
AI in Education: Rationale, Principles, and Instructional Implications
This study examines the integration of generative AI in schools, assessing its benefits and risks. As AI use by students grows, it's crucial to understand its impact on learning and teaching practices. Generative AI, like ChatGPT, can create human-like content, prompting questions about its educational role. The article differentiates large language models from traditional search engines and stresses the need for students to develop critical source evaluation skills. Although empirical evidence on AI's classroom effects is limited, AI offers personalized learning support and problem-solving tools, alongside challenges like undermining deep learning if misused. The study emphasizes deliberate strategies to ensure AI complements, not replaces, genuine cognitive effort. AI's educational role should be context-dependent, guided by pedagogical goals. The study concludes with practical advice for teachers on effectively utilizing AI to promote understanding and critical engagement, advocating for a balanced approach to enhance students' knowledge and skills development.
Rashomon effect in Educational Research: Why More is Better Than One for Measuring the Importance of the Variables?
Kuzilek, Jakub, Çavuş, Mustafa
This study explores how the Rashomon effect influences variable importance in the context of student demographics used for academic outcomes prediction. Our research follows the way machine learning algorithms are employed in Educational Data Mining, focusing on highlighting the so-called Rashomon effect. The study uses the Rashomon set of simple-yet-accurate models trained using decision trees, random forests, light GBM, and XGBoost algorithms with the Open University Learning Analytics Dataset. We found that the Rashomon set improves the predictive accuracy by 2-6%. Variable importance analysis revealed more consistent and reliable results for binary classification than multiclass classification, highlighting the complexity of predicting multiple outcomes. Key demographic variables imd_band and highest_education were identified as vital, but their importance varied across courses, especially in course DDD. These findings underscore the importance of model choice and the need for caution in generalizing results, as different models can lead to different variable importance rankings. The codes for reproducing the experiments are available in the repository: https://anonymous.4open.science/r/JEDM_paper-DE9D.
A multi-criteria decision support system to evaluate the effectiveness of training courses on citizens' employability
Bas, Maria C., Bolos, Vicente J., Prieto, Alvaro E., Rodriguez-Echeverria, Roberto, Sanchez-Figueroa, Fernando
This study examines the impact of lifelong learning on the professional lives of employed and unemployed individuals. Lifelong learning is a crucial factor in securing employment or enhancing one's existing career prospects. To achieve this objective, this study proposes the implementation of a multi-criteria decision support system for the evaluation of training courses in accordance with their capacity to enhance the employability of the students. The methodology is delineated in four stages. Firstly, a `working life curve' was defined to provide a quantitative description of an individual's working life. Secondly, an analysis based on K-medoids clustering defined a control group for each individual for comparison. Thirdly, the performance of a course according to each of the four predefined criteria was calculated using a t-test to determine the mean performance value of those who took the course. Ultimately, the unweighted TOPSIS method was used to evaluate the efficacy of the various training courses in relation to the four criteria. This approach effectively addresses the challenge of using extensive datasets within a system while facilitating the application of a multi-criteria unweighted TOPSIS method. The results of the multi-criteria TOPSIS method indicated that training courses related to the professional fields of administration and management, hostel and tourism and community and sociocultural services have positive impact on employability and improving the working conditions of citizens. However, courses that demonstrate the greatest effectiveness in ranking are the least demanded by citizens. The results will help policymakers evaluate the effectiveness of each training course offered by the regional government.