Active Learning for New Domains in Natural Language Understanding

ABSTRACT We explore active learning (AL) utterance selection for improving the accuracy of new underrepresented domains in a natural language understanding (NLU) system. Moreover, we propose an AL algorithm called Majority-CRF that uses an ensemble of classification and sequence labeling models to guide utterance selection for annotation. Experiments with three domains show that Majority-CRF achieves 6.6%-9% relative error rate reduction compared to random sampling with the same annotation budget, and statistically significant improvements compared to other AL approaches. Additionally, case studies with human-in-the-loop AL on six new domains show 4.6%-9% improvement on an existing NLU system. Index Terms-- Active Learning, Domain Expansion, Natural Language Understanding, Intelligent Virtual Assistants 1. INTRODUCTION Intelligent virtual assistants (IVA) with natural language understanding (NLU), such as Amazon Alexa, Apple Siri, Google Assistant, and Microsoft Cortana, are becoming increasingly popular. For IVA, NLU is a distinct component of spoken language understanding (SLU) [1], in conjunction with automatic speech recognition (ASR) and dialog management (DM). ASR produces a token sequence from speech, which is passed to NLU for both classifying the action or "intent" that the user wants to invoke (e.g.