A Reinforcement Learning Framework for Online Speaker Diarization

Speaker diarization is a crucial task in many real-world applications, such as meeting transcription, call center monitoring, and broadcast news processing. The goal of speaker diarization is to partition an audio or video stream into homogeneous segments, each corresponding to a single speaker, without any prior knowledge of the speakers' identities [1, 2]. This task has traditionally been addressed using unsupervised clustering methods [3, 4, 5], but recent advances in deep learning have led to the development of more powerful embedding-based approaches [6, 7, 5]. Despite the recent progress, speaker diarization remains a challenging problem, particularly in real-time and online scenarios where new speakers may enter or leave the conversation at any time. In such cases, pre-trained models may not be sufficient, and the system must be able to adapt to new speakers on the fly [8, 9, 10]. As in the successful applications to other speech and language tasks [11], the reinforcement learning (RL) has emerged as a promising approach for developing next-generation speaker diarization systems that can learn online and adapt to changing circumstances. In this paper, we propose a novel RL framework for online speaker diarization that does not require prior registration or pretraining. Our approach combines embedding extraction, clustering, and resegmentation into a single online decision-making problem, where the agent receives feedback in the form of rewards or penalties for each segmentation decision. We demonstrate the effectiveness of our approach using a Q-learning-based diarization agent on a desktop app, and discuss practical considerations for implementing and deploying RL-based speaker diarization systems.