There has recently been an explosion of work on spoken dialogue systems, along with an increased interest in open-domain systems that engage in casual conversations on popular topics such as movies, books and music. These systems aim to socially engage, entertain, and even empathize with their users. Since the achievement of such social goals is hard to measure, recent research has used dialogue length or human ratings as evaluation metrics, and developed methods for automatically calculating novel metrics, such as coherence, consistency, relevance and engagement. Here we develop a PARADISE model for predicting the performance of Athena, a dialogue system that has participated in thousands of conversations with real users, while competing as a finalist in the Alexa Prize. We use both user ratings and dialogue length as metrics for dialogue quality, and experiment with predicting these metrics using automatic features that are both system dependent and independent. Our goal is to learn a general objective function that can be used to optimize the dialogue choices of any Alexa Prize system in real time and evaluate its performance. Our best model for predicting user ratings gets an R$^2$ of .136 with a DistilBert model, and the best model for predicting length with system independent features gets an R$^2$ of .865, suggesting that conversation length may be a more reliable measure for automatic training of dialogue systems.

To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5 million dollar competition that challenges university teams to build conversational agents, or "socialbots", that can converse coherently and engagingly with humans on popular topics for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research at scale with real conversational data obtained by interacting with millions of Alexa users, along with user-provided ratings and feedback, over several months. This enables teams to effectively iterate, improve and evaluate their socialbots throughout the competition. Sixteen teams were selected for the inaugural competition last year. To build their socialbots, the students combined state-of-the-art techniques with their own novel strategies in the areas of Natural Language Understanding and Conversational AI.

Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user responses. The 2018 competition also included the provision of a suite of tools and models to the competitors including the CoBot (conversational bot) toolkit, topic and dialog act detection models, conversation evaluators, and a sensitive content detection model so that the competing teams could focus on building knowledge-rich, coherent and engaging multi-turn dialog systems. This paper outlines the advances developed by the university teams as well as the Alexa Prize team to achieve the common goal of advancing the science of Conversational AI. We address several key open-ended problems such as conversational speech recognition, open domain natural language understanding, commonsense reasoning, statistical dialog management and dialog evaluation. These collaborative efforts have driven improved experiences by Alexa users to an average rating of 3.61, median duration of 2 mins 18 seconds, and average turns to 14.6, increases of 14%, 92%, 54% respectively since the launch of the 2018 competition. For conversational speech recognition, we have improved our relative Word Error Rate by 55% and our relative Entity Error Rate by 34% since the launch of the Alexa Prize. Socialbots improved in quality significantly more rapidly in 2018, in part due to the release of the CoBot toolkit, with new entrants attaining an average rating of 3.35 just 1 week into the semifinals, compared to 9 weeks in the 2017 competition.

Conversational agents are exploding in popularity. However, much work remains in the area of non goal-oriented conversations, despite significant growth in research interest over recent years. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million dollar university competition where sixteen selected university teams built conversational agents to deliver the best social conversational experience. Alexa Prize provided the academic community with the unique opportunity to perform research with a live system used by millions of users. The subjectivity associated with evaluating conversations is key element underlying the challenge of building non-goal oriented dialogue systems. In this paper, we propose a comprehensive evaluation strategy with multiple metrics designed to reduce subjectivity by selecting metrics which correlate well with human judgement. The proposed metrics provide granular analysis of the conversational agents, which is not captured in human ratings. We show that these metrics can be used as a reasonable proxy for human judgment. We provide a mechanism to unify the metrics for selecting the top performing agents, which has also been applied throughout the Alexa Prize competition. To our knowledge, to date it is the largest setting for evaluating agents with millions of conversations and hundreds of thousands of ratings from users. We believe that this work is a step towards an automatic evaluation process for conversational AIs.

Eighteen teams were selected for the inaugural competition last year. To build their socialbots, the students combined state-of-the-art techniques with their own novel strategies in the areas of natural language understanding and conversational AI. This article reports on the research conducted over the 2017-2018 year. While the 20-minute grand challenge was not achieved in the first year, the competition produced several conversational agents that advanced the state of the art, that are interesting for everyday users to interact with, and that help form a baseline for the second year of the competition. We conclude with a summary of the human conversation have applicability in both work that we plan to address in the second year of professional and everyday domains. The first generation of such assistants -- Amazon's Alexa, Apple's Siri, Google The Alexa Prize competition received hundreds of Assistant, and Microsoft's Cortana -- have been applications from interested universities. After a focused on short, task-oriented interactions, such as detailed review of the applications, Amazon playing music or answering simple questions, as announced 12 sponsored and 6 unsponsored teams opposed to the longer free-form conversations that as the inaugural cohort for the Alexa Prize. The teams occur naturally in social and professional human that went live for the 2017 competition, listed alphabetically interaction. Conversational AI is the study of techniques by university, were DeisBot (Brandeis University), for creating software agents that can engage Magnus (Carnegie Mellon University), in natural conversational interactions with humans.

Onstage at the launch of Amazon's Alexa Prize, a multimillion-dollar competition to build AI that can chat like a human, the winners of last year's challenge delivered a friendly warning to 2018's hopefuls: your bot will mess up, it will say something offensive, and it will be taken offline. Elizabeth Clark, a member of last year's champion Sounding Board team from the University of Washington, was onstage with her fellow researchers to share what they'd learned from their experience. What stuck out, she said, were the bloopers. "One thing that came up a lot around the holidays was that a lot of people wanted to talk to our bot about Santa," said Clark. "Unfortunately, the content we had about Santa Claus looked like this: 'You know what I realized the other day? Santa Claus is the most elaborate lie ever told.'" The bot chose this line because it had been taught using jokes from Reddit, explained Clark, and while it might be diverting for adults, "as you can imagine, a lot of people who want to talk about Santa Claus … are children." And telling someone's curious three-year-old that Santa is a lie, right before Christmas? That's a conversational faux pas, even if you are just a dumb AI.

We present Sounding Board, a social chatbot that won the 2017 Amazon Alexa Prize. The system architecture consists of several components including spoken language processing, dialogue management, language generation, and content management, with emphasis on user-centric and content-driven design. We also share insights gained from large-scale online logs based on 160,000 conversations with real-world users.

The first interactor--a muscular man in his fifties with a shaved head and a black V-neck sweater--walks into a conference room and sits in a low-slung blue armchair before a phalanx of video cameras and studio lights. The rest of the room is totally dark. He gazes at a black, hockey- puck-shaped object--an Amazon Echo--on a small table in front of him. "Alexa," he says, "let's chat." "Good morning, my friend," a female voice replies with synthetic agreeability, a purplish ring of light pulsing atop the Echo.

Amazon's Alexa team is beginning to analyze the sound of users' voices to recognize their mood or emotional state, Alexa chief scientist Rohit Prasad told VentureBeat. Doing so could let Amazon personalize and improve customer experiences, lead to lengthier conversations with the AI assistant, and even open the door to Alexa one day responding to queries based on your emotional state or scanning voice recordings to diagnose disease. Tell Alexa that you're happy or sad today and she can deliver a pre-programmed response. In the future, Alexa may be able to pick up your mood without being told. The voice analysis effort will begin by teaching Alexa to recognize when a user is frustrated.

A Scottish university reached the final three of a prestigious international competition dedicated to advancing conversational artificial intelligence (AI). A team of Phd students from Heriot-Watt saw more than 100 entries from 22 countries including the likes of Harvard and Princeton to become the only UK institution to be placed in the Alexa Prize. A nine-strong team, named What's Up Bot, won plaudists from judges for their artificial intelligence software named Alana, which can understand and respond to human conversation. The annual competition is organised by online retail giant Amazon and is named after the Alexa voice command system that powers Amazon Echo. The team of PhD students from Heriot-Watt's school of mathematical and computer sciences finished behind fellow finalists, the Czech Technical University and eventual winners, the University of Washington, at a ceremony held in Las Vegas on Tuesday, 28 November.