Over the past few years the CES trade show has become a familiar post-holidays pilgrimage for many of the country's biggest marketers. They see the event as a way to get a sneak peek at the latest tech gadgets and technologies that can help them engage with their customers. This year marketing executives from companies such as Coca-Cola, Unilever, Johnson & Johnson, Campbell Soup and PepsiCo Inc. made their way to Las Vegas for the gathering. The convention was jam-packed with everything from self-driving cars to robots that play chess to Procter & Gamble's air-freshener spray that can connect with Alphabet Inc.'s Nest home to automatically release pleasant scents in the home. But there was one category that seemed to especially win over marketers: virtual assistants.
In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class.
Natural Language Processing (NLP) is a subfield of machine learning concerned with processing and analyzing natural language data, usually in the form of text or audio. Some common challenges within NLP include speech recognition, text generation, and sentiment analysis, while some high-profile products deploying NLP models include Apple's Siri, Amazon's Alexa, and many of the chatbots one might interact with online. To get started with NLP and introduce some of the core concepts in the field, we're going to build a model that tries to predict the sentiment (positive, neutral, or negative) of tweets relating to US Airlines, using the popular Twitter US Airline Sentiment dataset. Code snippets will be included in this post, but for fully reproducible notebooks and scripts, view all of the notebooks and scripts associated with this project on its Comet project page. Let's start by importing some libraries.