Sahay, Saurav
End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics
Okur, Eda, Sahay, Saurav, Alba, Roddy Fuentes, Nachman, Lama
The advances in language-based Artificial Intelligence (AI) technologies applied to build educational applications can present AI for social-good opportunities with a broader positive impact. Across many disciplines, enhancing the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages. Conversational AI systems have started maturing to a point where they could play a significant role in helping students learn fundamental math concepts. This work presents a task-oriented Spoken Dialogue System (SDS) built to support play-based learning of basic math concepts for early childhood education. The system has been evaluated via real-world deployments at school while the students are practicing early math concepts with multimodal interactions. We discuss our efforts to improve the SDS pipeline built for math learning, for which we explore utilizing MathBERT representations for potential enhancement to the Natural Language Understanding (NLU) module. We perform an end-to-end evaluation using real-world deployment outputs from the Automatic Speech Recognition (ASR), Intent Recognition, and Dialogue Manager (DM) components to understand how error propagation affects the overall performance in real-world scenarios.
Human in the loop approaches in multi-modal conversational task guidance system development
Manuvinakurike, Ramesh, Biswas, Sovan, Raffa, Giuseppe, Beckwith, Richard, Rhodes, Anthony, Shi, Meng, Mejia, Gesem Gudino, Sahay, Saurav, Nachman, Lama
Development of task guidance systems for aiding humans in a situated task remains a challenging problem. The role of search (information retrieval) and conversational systems for task guidance has immense potential to help the task performers achieve various goals. However, there are several technical challenges that need to be addressed to deliver such conversational systems, where common supervised approaches fail to deliver the expected results in terms of overall performance, user experience and adaptation to realistic conditions. In this preliminary work we first highlight some of the challenges involved during the development of such systems. We then provide an overview of existing datasets available and highlight their limitations. We finally develop a model-in-the-loop wizard-of-oz based data collection tool and perform a pilot experiment.
Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention
Su, Hsuan, Jhan, Jiun-Hao, Sun, Fan-yun, Sahay, Saurav, Lee, Hung-yi
Most chatbot literature that focuses on improving the fluency and coherence of a chatbot, is dedicated to making chatbots more human-like. However, very little work delves into what really separates humans from chatbots -- humans intrinsically understand the effect their responses have on the interlocutor and often respond with an intention such as proposing an optimistic view to make the interlocutor feel better. This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework includes a guiding chatbot and an interlocutor model that plays the role of humans. The guiding chatbot is assigned an intention and learns to induce the interlocutor to reply with responses matching the intention, for example, long responses, joyful responses, responses with specific words, etc. We examined our framework using three experimental setups and evaluated the guiding chatbot with four different metrics to demonstrate flexibility and performance advantages. Additionally, we performed trials with human interlocutors to substantiate the guiding chatbot's effectiveness in influencing the responses of humans to a certain extent. Code will be made available to the public.
Refine and Imitate: Reducing Repetition and Inconsistency in Persuasion Dialogues via Reinforcement Learning and Human Demonstration
Shi, Weiyan, Li, Yu, Sahay, Saurav, Yu, Zhou
Despite the recent success of large-scale language models on various downstream NLP tasks, the repetition and inconsistency problems still persist in dialogue response generation. Previous approaches have attempted to avoid repetition by penalizing the language model's undesirable behaviors in the loss function. However, these methods focus on token-level information and can lead to incoherent responses and uninterpretable behaviors. To alleviate these issues, we propose to apply reinforcement learning to refine an MLE-based language model without user simulators, and distill sentence-level information about repetition, inconsistency and task relevance through rewards. In addition, to better accomplish the dialogue task, the model learns from human demonstration to imitate intellectual activities such as persuasion, and selects the most persuasive responses. Experiments show that our model outperforms previous state-of-the-art dialogue models on both automatic metrics and human evaluation results on a donation persuasion task, and generates more diverse, consistent and persuasive conversations according to the user feedback.
Intentional Analysis of Medical Conversations for Community Engagement
Sahay, Saurav (Georgia Institute of Technology)
With an explosion in the proliferation of user-generated content in communities, information overload is increasing and quality of readily available online content is deteriorating. There is an increasing need for intelligent systems that make use of implicit user generated knowledge in communities for community engagement. We describe our approach based on modeling user utterances in communities to proactively target the community for exchange of questions and answers. We envision a system that automatically encourages user engagement and participation by routing relevant conversations to users based on individual and community activity levels. In this paper, we analyze health forum conversations from WebMD, a popular health portal consumer site, and classify them in different acts of speech using Verbal Response Modes (VRM) theory. We describe our approach for modeling an intelligent community recommender to engage participants based on observations from our analysis.
Socio-Semantic Health Information Access
Sahay, Saurav (Georgia Institute of Technology) | Ram, Ashwin (Georgia Institute of Technology)
We describe Cobot, a mixed initiative socio-semantic conversational search and recommendation system for finding health information. With Cobot, users can start a real time conversation about their health concerns. Cobot then connects relevant users together in the conversation also providing contextual recommendations relevant to the conversation. Conventional search engines and content portals provide a solitary search experience inundating the health information seeker with a hoard of information often confusing and frustrating them. Cobot brings relevant healthcare information directly or through other users without any search through natural language conversation.