heffernan
Can Large Language Models Replicate ITS Feedback on Open-Ended Math Questions?
McNichols, Hunter, Lee, Jaewook, Fancsali, Stephen, Ritter, Steve, Lan, Andrew
Intelligent Tutoring Systems (ITSs) often contain an automated feedback component, which provides a predefined feedback message to students when they detect a predefined error. To such a feedback component, we often resort to template-based approaches. These approaches require significant effort from human experts to detect a limited number of possible student errors and provide corresponding feedback. This limitation is exemplified in open-ended math questions, where there can be a large number of different incorrect errors. In our work, we examine the capabilities of large language models (LLMs) to generate feedback for open-ended math questions, similar to that of an established ITS that uses a template-based approach. We fine-tune both open-source and proprietary LLMs on real student responses and corresponding ITS-provided feedback. We measure the quality of the generated feedback using text similarity metrics. We find that open-source and proprietary models both show promise in replicating the feedback they see during training, but do not generalize well to previously unseen student errors. These results suggest that despite being able to learn the formatting of feedback, LLMs are not able to fully understand mathematical errors made by students.
- North America > United States > Massachusetts > Hampshire County > Amherst (0.04)
- North America > Canada > Ontario > Toronto (0.04)
- Europe > Switzerland (0.04)
- (2 more...)
- Education > Educational Setting (0.68)
- Education > Educational Technology > Educational Software > Computer Based Training (0.55)
Interpreting Deep Knowledge Tracing Model on EdNet Dataset
Wang, Deliang, Lu, Yu, Meng, Qinggang, Chen, Penghe
With more deep learning techniques being introduced into the knowledge tracing domain, the interpretability issue of the knowledge tracing models has aroused researchers' attention. Our previous study(Lu et al. 2020) on building and interpreting the KT model mainly adopts the ASSISTment dataset(Feng, Heffernan, and Koedinger 2009),, whose size is relatively small. In this work, we perform the similar tasks but on a large and newly available dataset, called EdNet(Choi et al. 2020). The preliminary experiment results show the effectiveness of the interpreting techniques, while more questions and tasks are worthy to be further explored and accomplished.
- Asia > China > Beijing > Beijing (0.05)
- North America > United States > Pennsylvania > Allegheny County > Pittsburgh (0.04)
- Asia > Singapore (0.04)
Chatbots Allow Educators to Delegate Repetitive Tasks and Focus on Teaching
Colleges have had success with chatbots for a few years, but high school students can now benefit from the first nationally accessible (and free) AI college adviser chatbot, Oli. The tool is the result of a partnership between Common App and Mainstay (formerly AdmitHub). Oli stands ready to help students around the clock with a wide range of tasks, such as selecting the right school, completing college and scholarship applications and understanding financial aid forms. It responds to questions via text and also sends users deadline reminders, updates and resources several times a week. When extra help is required, Oli connects students with a trained college adviser from College Advising Corps.
MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
Shen, Jia Tracy, Yamashita, Michiharu, Prihar, Ethan, Heffernan, Neil, Wu, Xintao, Lee, Dongwon
Due to the transfer learning nature of BERT model, researchers have achieved better performance than base BERT by further pre-training the original BERT on a huge domain-specific corpus. Due to the special nature of mathematical texts which often contain math equations and symbols, the original BERT model pre-trained on general English context will not fit Natural Language Processing (NLP) tasks in mathematical education well. Therefore, we propose MathBERT, a BERT pre-trained on large mathematical corpus including pre-k to graduate level mathematical content to tackle math-specific tasks. In addition, We generate a customized mathematical vocabulary to pre-train with MathBERT and compare the performance to the MathBERT pre-trained with the original BERT vocabulary. We select three important tasks in mathematical education such as knowledge component, auto-grading, and knowledge tracing prediction to evaluate the performance of MathBERT. Our experiments show that MathBERT outperforms the base BERT by 2-9\% margin. In some cases, MathBERT pre-trained with mathematical vocabulary is better than MathBERT trained with original vocabulary.To our best knowledge, MathBERT is the first pre-trained model for general purpose mathematics education tasks.
- Education > Educational Technology > Educational Software (0.47)
- Education > Educational Setting > Online (0.46)
- Education > Curriculum > Subject-Specific Education (0.35)
pyBKT: An Accessible Python Library of Bayesian Knowledge Tracing Models
Badrinath, Anirudhan, Wang, Frederic, Pardos, Zachary
Bayesian Knowledge Tracing, a model used for cognitive mastery estimation, has been a hallmark of adaptive learning research and an integral component of deployed intelligent tutoring systems (ITS). In this paper, we provide a brief history of knowledge tracing model research and introduce pyBKT, an accessible and computationally efficient library of model extensions from the literature. The library provides data generation, fitting, prediction, and cross-validation routines, as well as a simple to use data helper interface to ingest typical tutor log dataset formats. We evaluate the runtime with various dataset sizes and compare to past implementations. Additionally, we conduct sanity checks of the model using experiments with simulated data to evaluate the accuracy of its EM parameter learning and use real-world data to validate its predictions, comparing pyBKT's supported model variants with results from the papers in which they were originally introduced. The library is open source and open license for the purpose of making knowledge tracing more accessible to communities of research and practice and to facilitate progress in the field through easier replication of past approaches.
- Research Report (0.50)
- Instructional Material (0.47)
- Information Technology > Software > Programming Languages (1.00)
- Information Technology > Artificial Intelligence > Representation & Reasoning (1.00)
- Information Technology > Artificial Intelligence > Machine Learning > Performance Analysis (0.67)
- Information Technology > Artificial Intelligence > Machine Learning > Learning Graphical Models > Directed Networks > Bayesian Learning (0.46)
Covid crisis shifts supply chain management from efficiency to resilience
Looked at on a world scale, the Covid-19 pandemic will continue to deliver shocks to global supply chains for some time to come. Even if the public health crisis abates in the UK, our economy is part of a global economy, and UK corporate IT will have its work cut out in supporting companies as they are forced to re-forge supply chains, perhaps over and over again, and at short notice. The crisis has provoked some rethinking of how the world economy ought to work, with an emphasis on the desirability of a shift from efficiency – doing things "just in time" – to resilience – building in more slack. The FT's Rana Faroohar provides an account of such rethinking in an article entitled From'just in time' to'just in case' published earlier this year. In the discussions which lie behind this article there are different emphases on a spectrum of opinion: some say we can have both efficiency and resilience equally, others that there is a choice to be made for one or the other, and yet others say it's a matter of balance, of trading off. Tony Harris, global vice-president of business network solutions at SAP, says it has to be a combination. "You wouldn't want to move to a resilient network or supply chain that wasn't also efficient," he says.
- Europe > United Kingdom (0.34)
- Asia > China (0.06)
- South America > Brazil (0.04)
- (7 more...)
- Banking & Finance > Economy (1.00)
- Health & Medicine > Therapeutic Area > Infections and Infectious Diseases (0.50)
- Health & Medicine > Therapeutic Area > Immunology (0.50)
WQED's "The Robot Doctor" Brings CMU Expertise to PA High School Students
What do you picture when you think of a robot? That's the first question asked by "The Robot Doctor" -- a new series created by Carnegie Mellon University educators, RobotWits, the Pennsylvania Rural Robotics Initiative and WQED. Airing on PBS stations across Pennsylvania, the eight-episode program is geared toward high school students who may lack access to a computer during school closures, and who live in underresourced areas with limited STEM opportunities. "We're going to explore how robots solve the problems that allow them to be useful in the world. We'll do this with nothing more than the math concepts you may already know: geometry, trigonometry, basic algebra and a few concepts from physics," Jonathan Butzke says in the first episode. Butzke, an alumnus of CMU's Robotics Institute, hosts the show and is lead robotics researcher for RobotWits.
Under the skin: how insertable microchips could unlock the future
The microchip is about the size of a grain of rice and usually inserted in the webbing between the thumb and forefinger using a needle the same thickness as used in body piercing. It feels, says insertable technology expert Kayla Heffernan, like getting a drip. Once the needle is removed the incision heals in a few days and the microchip remains, allowing the wearer to open doors with the brush of a hand – provided they only wish to access one particular place. Commercially available insertable microchips are only large enough to hold one access code and a small amount of other information, so the days of replacing an entire wallet and keychain with a tiny computer under the skin are not yet upon us. The future is coming, but it's not in a rush.
- North America > United States > Wisconsin (0.05)
- Europe > Sweden > Stockholm > Stockholm (0.05)
Demystify the technology that creates AI
Beware of relying uncritically on big data computer systems, warns a St. Mary's University professor undertaking a five-year research project dubbed Where Science Meets Fiction: Social Robots and the Ethical Imagination. "There are real dangers now with big data," said Dr. Teresa Heffernan, the St. Mary's University professor undertaking the research project. "Algorithms have the same biases as humans." With her research project, the professor is hoping to demystify the technology that creates artificial intelligence and bring together experts from all walks of life to begin a dialogue about how humans and these machines should interact -- what to do and what not to do. "I want to shift the conversation that has been shaped by Silicon Valley . . . to make it more open and question the rhetoric, to demystify the technology and expose how the technology works rather than be dominated by it," said Heffernan.
- North America > United States > California (0.25)
- North America > Canada (0.05)
- Europe > Switzerland (0.05)
- Information Technology (0.89)
- Banking & Finance > Economy (0.52)
How Computerized Tutors Are Learning to Teach Humans
Neil Heffernan was listening to his fiancée, Cristina Lindquist, tutor one of her students in mathematics when he had an idea. Heffernan was a graduate student in computer science, and by this point -- the summer of 1997 -- he had been working for two years with researchers at Carnegie Mellon University on developing computer software to help students improve their skills. But he had come to believe that the programs did little to assist their users. They were built on elaborate theories of the student mind -- attempts to simulate the learning brain. Then it dawned on him: what was missing from the programs was the interventions teachers made to promote and accelerate learning.