The popularity of conversational digital assistants has resulted in the availability of large amounts of conversational data which can be utilized for improved user experience and personalized response generation. Building these assistants using popular large language models like ChatGPT also require additional emphasis on prompt engineering and evaluation methods. Textual similarity metrics are a key ingredient for such analysis and evaluations. While many similarity metrics have been proposed in the literature, they have not proven effective for task-oriented conversations as they do not take advantage of unique conversational features. To address this gap, we present TaskDiff, a novel conversational similarity metric that utilizes different dialogue components (utterances, intents, and slots) and their distributions to compute similarity. Extensive experimental evaluation of TaskDiff on a benchmark dataset demonstrates its superior performance and improved robustness over other related approaches.

Follow up on my previous post discussing the key technologies around the conversational AI solution, I will be dive into the typical challenges the AI Engineer team would encounter when building a virtual agent or a chatbot solution for your clients or customers. Let firstly define the scope and goal of the conversational application. The conversational agents can be categorized into two main streams. The typical agents for Open Domain Conversation are Siri, Google Assistant, BlenderBot from Facebook, Meena from Google. Users can start a conversation without a clear goal, and the topics are unrestricted.