Additional Shared Decoder on Siamese Multi-view Encoders for Learning Acoustic Word Embeddings

ADDITIONAL SHARED DECODER ON SIAMESE MUL TI-VIEW ENCODERS FOR LEARNING ACOUSTIC WORD EMBEDDINGS Myunghun Jung, Hyungjun Lim, Jahyun Goo, Y oungmoon Jung, and Hoirin Kim School of Electrical Engineering, KAIST, Daejeon, Republic of Korea ABSTRACT Acoustic word embeddings -- fixed-dimensional vector representations of arbitrary-length words -- have attracted increasing interest in query-by-example spoken term detection. Recently, on the fact that the orthography of text labels partly reflects the phonetic similarity between the words' pronunciation, a multi-view approach has been introduced that jointly learns acoustic and text embeddings. It showed that it is possible to learn discriminative embeddings by designing the objective which takes text labels as well as word segments. In this paper, we propose a network architecture that expands the multi-view approach by combining the Siamese multi-view encoders with a shared decoder network to maximize the effect of the relationship between acoustic and text em-beddings in embedding space. Discriminatively trained with multi-view triplet loss and decoding loss, our proposed approach achieves better performance on acoustic word discrimination task with the WSJ dataset, resulting in 11.1% relative improvement in average precision. Index T erms -- acoustic word embedding, query-by- example spoken term detection, multi-view learning, Siamese network, encoder-decoder 1. INTRODUCTION Query-by-example spoken term detection (QbE-STD) is the task of retrieving a spoken query from a set of speech utterances. Amazon Echo, Google Home, Apple Siri), the QbE-STD has drawn interest as a technique that can be applied to wake-up or command word detection, search engine, etc.