Discourse & Dialogue
Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model
Chen, Xiaolin, Song, Xuemeng, Jing, Liqiang, Li, Shuo, Hu, Linmei, Nie, Liqiang
Text response generation for multimodal task-oriented dialog systems, which aims to generate the proper text response given the multimodal context, is an essential yet challenging task. Although existing efforts have achieved compelling success, they still suffer from two pivotal limitations: 1) overlook the benefit of generative pre-training, and 2) ignore the textual context related knowledge. To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language model for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection, dual knowledge-enhanced context learning, and knowledge-enhanced response generation. To be specific, the dual knowledge selection component aims to select the related knowledge according to both textual and visual modalities of the given context. Thereafter, the dual knowledge-enhanced context learning component targets seamlessly integrating the selected knowledge into the multimodal context learning from both global and local perspectives, where the cross-modal semantic relation is also explored. Moreover, the knowledge-enhanced response generation component comprises a revised BART decoder, where an additional dot-product knowledge-decoder attention sub-layer is introduced for explicitly utilizing the knowledge to advance the text response generation. Extensive experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.
BiSyn-GAT+: Bi-Syntax Aware Graph Attention Network for Aspect-based Sentiment Analysis
Liang, Shuo, Wei, Wei, Mao, Xian-Ling, Wang, Fei, He, Zhiyong
Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis task that aims to align aspects and corresponding sentiments for aspect-specific sentiment polarity inference. It is challenging because a sentence may contain multiple aspects or complicated (e.g., conditional, coordinating, or adversative) relations. Recently, exploiting dependency syntax information with graph neural networks has been the most popular trend. Despite its success, methods that heavily rely on the dependency tree pose challenges in accurately modeling the alignment of the aspects and their words indicative of sentiment, since the dependency tree may provide noisy signals of unrelated associations (e.g., the "conj" relation between "great" and "dreadful" in Figure 2). In this paper, to alleviate this problem, we propose a Bi-Syntax aware Graph Attention Network (BiSyn-GAT+). Specifically, BiSyn-GAT+ fully exploits the syntax information (e.g., phrase segmentation and hierarchical structure) of the constituent tree of a sentence to model the sentiment-aware context of every single aspect (called intra-context) and the sentiment relations across aspects (called inter-context) for learning. Experiments on four benchmark datasets demonstrate that BiSyn-GAT+ outperforms the state-of-the-art methods consistently.
Emotion Recognition in Conversation using Probabilistic Soft Logic
Augustine, Eriq, Jandaghi, Pegah, Albalak, Alon, Pryor, Connor, Dickens, Charles, Wang, William, Getoor, Lise
Creating agents that can both appropriately respond to conversations and understand complex human linguistic tendencies and social cues has been a long standing challenge in the NLP community. A recent pillar of research revolves around emotion recognition in conversation (ERC); a sub-field of emotion recognition that focuses on conversations or dialogues that contain two or more utterances. In this work, we explore an approach to ERC that exploits the use of neural embeddings along with complex structures in dialogues. We implement our approach in a framework called Probabilistic Soft Logic (PSL), a declarative templating language that uses first-order like logical rules, that when combined with data, define a particular class of graphical model. Additionally, PSL provides functionality for the incorporation of results from neural models into PSL models. This allows our model to take advantage of advanced neural methods, such as sentence embeddings, and logical reasoning over the structure of a dialogue. We compare our method with state-of-the-art purely neural ERC systems, and see almost a 20% improvement. With these results, we provide an extensive qualitative and quantitative analysis over the DailyDialog conversation dataset.
A Flexible Schema-Guided Dialogue Management Framework: From Friendly Peer to Virtual Standardized Cancer Patient
Kane, Benjamin, Giugno, Catherine, Schubert, Lenhart, Haut, Kurtis, Wohn, Caleb, Hoque, Ehsan
A schema-guided approach to dialogue management has been shown in recent work to be effective in creating robust customizable virtual agents capable of acting as friendly peers or task assistants. However, successful applications of these methods in open-ended, mixed-initiative domains remain elusive -- particularly within medical domains such as virtual standardized patients, where such complex interactions are commonplace -- and require more extensive and flexible dialogue management capabilities than previous systems provide. In this paper, we describe a general-purpose schema-guided dialogue management framework used to develop SOPHIE, a virtual standardized cancer patient that allows a doctor to conveniently practice for interactions with patients. We conduct a crowdsourced evaluation of conversations between medical students and SOPHIE. Our agent is judged to produce responses that are natural, emotionally appropriate, and consistent with her role as a cancer patient. Furthermore, it significantly outperforms an end-to-end neural model fine-tuned on a human standardized patient corpus, attesting to the advantages of a schema-guided approach.
2020 U.S. presidential election in swing states: Gender differences in Twitter conversations
Karami, Amir, Clark, Spring B., Mackenzie, Anderson, Lee, Dorathea, Zhu, Michael, Boyajieff, Hannah R., Goldschmidt, Bailey
Social media is commonly used by the public during election campaigns to express their opinions regarding different issues. Among various social media channels, Twitter provides an efficient platform for researchers and politicians to explore public opinion regarding a wide range of topics such as the economy and foreign policy. Current literature mainly focuses on analyzing the content of tweets without considering the gender of users. This research collects and analyzes a large number of tweets and uses computational, human coding, and statistical analyses to identify topics in more than 300,000 tweets posted during the 2020 U.S. presidential election and to compare female and male users regarding the average weight of the discussed topics. Our findings are based upon a wide range of topics, such as tax, climate change, and the COVID-19 pandemic. Out of the topics, there exists a significant difference between female and male users for more than 70% of topics.
Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems
Razumovskaia, Evgeniia (Language Technology Lab, University of Cambridge, UK) | Glavas, Goran (Data and Web Science Group, University of Mannheim, Germany) | Majewska, Olga (Language Technology Lab, University of Cambridge, UK) | Ponti, Edoardo M. (Mila - Quebec AI Institute and McGill University, Canada) | Korhonen, Anna (University of Cambridge, UK) | Vulic, Ivan (Language Technology Lab, University of Cambridge, UK)
In task-oriented dialogue (ToD), a user holds a conversation with an artificial agentย with the aim of completing a concrete task. Although this technology represents one ofย the central objectives of AI and has been the focus of ever more intense research andย development efforts, it is currently limited to a few narrow domains (e.g., food ordering,ย ticket booking) and a handful of languages (e.g., English, Chinese). This work provides anย extensive overview of existing methods and resources in multilingual ToD as an entry pointย to this exciting and emerging field. We find that the most critical factor preventing theย creation of truly multilingual ToD systems is the lack of datasets in most languages forย both training and evaluation. In fact, acquiring annotations or human feedback for eachย component of modular systems or for data-hungry end-to-end systems is expensive andย tedious. Hence, state-of-the-art approaches to multilingual ToD mostly rely on (zero- orย few-shot) cross-lingual transfer from resource-rich languages (almost exclusively English),ย either by means of (i) machine translation or (ii) multilingual representations. Theseย approaches are currently viable only for typologically similar languages and languages withย parallel / monolingual corpora available. On the other hand, their effectiveness beyond theseย boundaries is doubtful or hard to assess due to the lack of linguistically diverse benchmarksย (especially for natural language generation and end-to-end evaluation). To overcome thisย limitation, we draw parallels between components of the ToD pipeline and other NLP tasks,ย which can inspire solutions for learning in low-resource scenarios. Finally, we list additionalย challenges that multilinguality poses for related areas (such as speech, fluency in generatedย text, and human-centred evaluation), and indicate future directions that hold promise toย further expand language coverage and dialogue capabilities of current ToD systems.ย
An Introduction to Lifelong Supervised Learning
Sodhani, Shagun, Faramarzi, Mojtaba, Mehta, Sanket Vaibhav, Malviya, Pranshu, Abdelsalam, Mohamed, Janarthanan, Janarthanan, Chandar, Sarath
This primer is an attempt to provide a detailed summary of the different facets of lifelong learning. We start with Chapter 2 which provides a high-level overview of lifelong learning systems. In this chapter, we discuss prominent scenarios in lifelong learning (Section 2.4), provide 8 Introduction a high-level organization of different lifelong learning approaches (Section 2.5), enumerate the desiderata for an ideal lifelong learning system (Section 2.6), discuss how lifelong learning is related to other learning paradigms (Section 2.7), describe common metrics used to evaluate lifelong learning systems (Section 2.8). This chapter is more useful for readers who are new to lifelong learning and want to get introduced to the field without focusing on specific approaches or benchmarks. The remaining chapters focus on specific aspects (either learning algorithms or benchmarks) and are more useful for readers who are looking for specific approaches or benchmarks. Chapter 3 focuses on regularization-based approaches that do not assume access to any data from previous tasks. Chapter 4 discusses memory-based approaches that typically use a replay buffer or an episodic memory to save subset of data across different tasks. Chapter 5 focuses on different architecture families (and their instantiations) that have been proposed for training lifelong learning systems. Following these different classes of learning algorithms, we discuss the commonly used evaluation benchmarks and metrics for lifelong learning (Chapter 6) and wrap up with a discussion of future challenges and important research directions in Chapter 7.
Twitmo: A Twitter Data Topic Modeling and Visualization Package for R
Buchmรผller, Andreas, Kant, Gillian, Weisser, Christoph, Sรคfken, Benjamin, Kis-Katos, Krisztina, Kneib, Thomas
We present Twitmo, a package that provides a broad range of methods to collect, pre-process, analyze and visualize geo-tagged Twitter data. Twitmo enables the user to collect geo-tagged Tweets from Twitter and and provides a comprehensive and user-friendly toolbox to generate topic distributions from Latent Dirichlet Allocations (LDA), correlated topic models (CTM) and structural topic models (STM). Functions are included for pre-processing of text, model building and prediction. In addition, one of the innovations of the package is the automatic pooling of Tweets into longer pseudo-documents using hashtags and cosine similarities for better topic coherence. The package additionally comes with functionality to visualize collected data sets and fitted models in static as well as interactive ways and offers built-in support for model visualizations via LDAvis providing great convenience for researchers in this area. The Twitmo package is an innovative toolbox that can be used to analyze public discourse of various topics, political parties or persons of interest in space and time.
SETSum: Summarization and Visualization of Student Evaluations of Teaching
Hu, Yinuo, Zhang, Shiyue, Sathy, Viji, Panter, A. T., Bansal, Mohit
Student Evaluations of Teaching (SETs) are widely used in colleges and universities. Typically SET results are summarized for instructors in a static PDF report. The report often includes summary statistics for quantitative ratings and an unsorted list of open-ended student comments. The lack of organization and summarization of the raw comments hinders those interpreting the reports from fully utilizing informative feedback, making accurate inferences, and designing appropriate instructional improvements. In this work, we introduce a novel system, SETSum, that leverages sentiment analysis, aspect extraction, summarization, and visualization techniques to provide organized illustrations of SET findings to instructors and other reviewers. Ten university professors from diverse departments serve as evaluators of the system and all agree that SETSum helps them interpret SET results more efficiently; and 6 out of 10 instructors prefer our system over the standard static PDF report (while the remaining 4 would like to have both). This demonstrates that our work holds the potential to reform the SET reporting conventions in the future. Our code is available at https://github.com/evahuyn/SETSum