Large Language Models (LLMs) are one of the most promising technologies for the next era of speech generation systems, due to their scalability and in-context learning capabilities. Nevertheless, they suffer from multiple stability issues at inference time, such as hallucinations, content skipping or speech repetitions. In this work, we introduce a new self-supervised Voice Conversion (VC) architecture which can be used to learn to encode transitory features, such as content, separately from stationary ones, such as speaker ID or recording conditions, creating speaker-disentangled representations. Using speaker-disentangled codes to train LLMs for text-to-speech (TTS) allows the LLM to generate the content and the style of the speech only from the text, similarly to humans, while the speaker identity is provided by the decoder of the VC model. Results show that LLMs trained over speaker-disentangled self-supervised representations provide an improvement of 4.7pp in speaker similarity over SOTA entangled representations, and a word error rate (WER) 5.4pp lower. Furthermore, they achieve higher naturalness than human recordings of the LibriTTS test-other dataset. Finally, we show that using explicit reference embedding negatively impacts intelligibility (stability), with WER increasing by 14pp compared to the model that only uses text to infer the style.

A popular approach consists in recording a smaller dataset featuring the desired emphasis effect in addition to the main We present a scalable method to produce high quality emphasis'neutral' recordings, and having the model learn the particular for text-to-speech (TTS) that does not require recordings or prosody associated with the emphasized words (see [5, 6, 7, 8] annotations. Many TTS models include a phoneme duration for recent examples). We build one such model as our upper model. A simple but effective method to achieve emphasized anchor, as detailed in section 2.1 speech consists in increasing the predicted duration of the emphasised While this technique works well for the speaker for which word. We show that this is significantly better than'emphasis recordings' are available, it does not directly scale spectrogram modification techniques improving naturalness by to new speakers or different languages. An alternative technique 7.3% and correct testers' identification of the emphasized word adopted with varying degrees of success consists in annotating in a sentence by 40% on a reference female en-US voice.