On the Effectiveness of Acoustic BPE in Decoder-Only TTS

To address this issue, one possible way is to further compress the discrete speech sequence. A promising approach is Discretizing speech into tokens and generating them by a the acoustic byte-pair encoding (BPE) technique, which is proposed decoder-only model have been a promising direction for text-tospeech in [15]. It is a similar method to the traditional BPE algorithm (TTS) and spoken language modeling (SLM). To shorten [16] in natural language processing. It treats the discrete the sequence length of speech tokens, acoustic byte-pair encoding indexes of speech as literal characters and iteratively compresses (BPE) has emerged in SLM that treats speech tokens from consecutive tokens based on the frequency in the training self-supervised semantic representations as characters to further corpus. Such compression will coherently reduce sequence compress the token sequence. But the gain in TTS has not been length with the increase of vocabulary size. For speech discrete fully investigated, and the proper choice of acoustic BPE remains tokens, usually a group of multiple tokens occur together to represent unclear. In this work, we conduct a comprehensive study a specific phoneme or syllable, and organizing them to on various settings of acoustic BPE to explore its effectiveness be a unique modeling unit would provide a higher abstraction in decoder-only TTS models with semantic speech tokens.