Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions

Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions Jiwon Suh, Injae Na, W oohwan Jung Department of Applied Artificial Intelligence, Hanyang University, Republic of Korea {jwsuh0205, suhoij47, whjung}@hanyang.ac.kr Abstract End-to-end automatic speech recognition (E2E ASR) systems have significantly improved speech recognition through training on extensive datasets. Despite these advancements, they still struggle to accurately recognize domain specific words, such as proper nouns and technical terminologies. To address this problem, we propose a method to utilize the state-of-the-art Whisper without modifying its architecture, preserving its generalization performance while enabling it to leverage descriptions effectively. Moreover, we propose two additional training techniques to improve the domain specific ASR: decoder fine-tuning, and context perturbation. We also propose a method to use a Large Language Model (LLM) to generate descriptions with simple metadata, when descriptions are unavailable. Our experiments demonstrate that proposed methods notably enhance domain-specific ASR accuracy on real-life datasets, with LLMgenerated descriptions outperforming human-crafted ones in effectiveness. Introduction Recent advancements in end-to-end (E2E) automatic speech recognition (ASR) systems, such as Wav2V ec 2.0 [1] and Whisper [2], have significantly improved the capabilities of speech recognition through extensive training on large datasets. However, these systems often encounter difficulties in accurately identifying domain specific terms, such as proper nouns and technical jargon.