Double Mixture: Towards Continual Event Detection from Speech

Speech event detection is crucial for multimedia retrieval, involving the tagging of both semantic and acoustic events. Traditional ASR systems often overlook the interplay between these events, focusing solely on content, even though the interpretation of dialogue can vary with environmental context. This paper tackles two primary challenges in speech event detection: the continual integration of new events without forgetting previous ones, and the disentanglement of semantic from acoustic events. We introduce a new task, continual event detection from speech, for which we also provide two benchmark datasets. To address the challenges of catastrophic forgetting and effective disentanglement, Figure 1: In continual learning, learners incrementally acquire we propose a novel method, 'Double Mixture.' This method merges new event types and must evaluate all previously speech expertise with robust memory mechanisms to enhance learned types during testing. This process is particularly adaptability and prevent forgetting. Our comprehensive experiments challenging in speech-based scenarios due to the complex interplay show that this task presents significant challenges that are of semantic content (semantic event) and background not effectively addressed by current state-of-the-art methods in either sounds (acoustic event).