Direct optimization of F-measure for retrieval-based personal question answering

DIRECT OPTIMIZA TION OF F-MEASURE FOR RETRIEV AL-BASED PERSONAL QUESTION ANSWERING Rasool Fakoor†, Amanjit Kainth, Siamak Shakeri, Christopher Winestock, Abdel-rahman Mohamed, Ruhi Sarikaya Amazon ABSTRACT Recent advances in spoken language technologies and the introduction of many customer facing products, have given rise to a wide customer reliance on smart personal assistants for many of their daily tasks. In this paper, we present a system to reduce users' cognitive load by extending personal assistants with long-term personal memory where users can store and retrieve by voice, arbitrary pieces of information. The problem is framed as a neural retrieval based question answering system where answers are selected from previously stored user memories. We propose to directly optimize the end-to-end retrieval performance, measured by the F1-score, using reinforcement learning, leading to better performance on our experimental test set(s). Index Terms-- Question Answering, Spoken information retrieval, Reinforcement Learning, Personal Assistants 1. INTRODUCTION Recent advances in speech recognition [1, 2], speech enhancement [3, 4], natural language understanding [5, 6], question answering [7, 8, 9], and dialogue systems [10, 11] have fueled the current surge in research and development for smart personal assistants [12] like Alexa, Siri, Google assistant, and Cortana, with many use cases around shopping, music, etc. In this paper we present a system for providing personal assistants a long term personal memory that enable users to store anything they want to remember by voice, and then later ask questions about it. An example use case is shown in Table 1.