Bimodal Speech Emotion Recognition Using Pre-Trained Language Models

ABSTRACT Speech emotion recognition is a challenging task and an important step towards more natural human-machine interaction. We show that pre-trained language models can be fine-tuned for text emotion recognition, achieving an accuracy of 69 .5 % on Task 4A of SemEval 2017, improving upon the previous state of the art by over 3 % absolute. We combine these language models with speech emotion recognition, achieving results of 73. 5 % accuracy when using provided transcriptions and speech data on a subset of four classes of the IEMOCAP dataset. For our experiments, we created IEmoNet, a modular and adaptable bimodal framework for speech emotion recognition based on pre-trained language models. Lastly, we discuss the idea of using an emotional classifier as a reward for reinforcement learning as a step towards more successful and convenient human-machine interaction. Index T erms-- Speech Emotion Recognition, Text Emotion Recognition, Bimodal Emotion Recognition, IEMOCAP, Self Attention, Pre-trained Language Models 1. INTRODUCTION Emotions are an important aspect of human behavior. They do not only influence the reaction to our environment [1, 2], but also actively change our perception of it [3] and sometimes even contribute to how well we remember specific events [4]. As such, they influence both human-human and human-machine interaction. However, in human-machine interaction, emotions are often not at all or only scarcely considered.