Hybrid Emotion Recognition: Enhancing Customer Interactions Through Acoustic and Textual Analysis

Sahan Hewage Wewelwala School of Computing Informatics Institute of Technology Colombo 06, Sri Lanka sahanwewelwala@gmail.com T.G.D.K. Sumanathilaka Department of Computer Science Swansea University Swansea, Wales, United Kingdom deshankoshala@gmail.com Abstract -- This research presents a hybrid emotion recognition system integrating advanced Deep Learning, Natural Language Processing (NLP), and Large Language Models (LLMs) to analyze audio and textual data for enhancing customer interactions in contact centers. By combining acoustic features with textual sentiment analysis, the system achieves nuanced emotion detection, addressing the limitations of traditional approaches in understanding complex emotional states. Rigorous testing on diverse datasets demonstrates the system's robustness and accuracy, highlighting its potential to transform customer service by enabling personalized, empathetic interactions and improving operational efficiency. This research establishes a foundation for more intelligent and human - centric digital communication, redefining customer service standards. The capacity to identify and comprehend emotions effectively is an essential element of human - computer interaction, especially in spoken and written communication.