We present MELLE, a novel continuous-valued tokens based language modeling approach for text to speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which are originally designed for audio compression and sacrifice fidelity compared to mel-spectrograms. Specifically, (i) instead of cross-entropy loss, we apply regression loss with a proposed spectrogram flux loss function to model the probability distribution of the continuous-valued tokens. (ii) we have incorporated variational inference into MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity and model robustness. Experiments demonstrate that, compared to the two-stage codec language models VALL-E and its variants, the single-stage MELLE mitigates robustness issues by avoiding the inherent flaws of sampling discrete codes, achieves superior performance across multiple metrics, and, most importantly, offers a more streamlined paradigm. See https://aka.ms/melle for demos of our work.

It's the fingers that get to me. Bitten and weathered, the skin worn down in places, they are incredibly detailed and utterly convincing. I'm looking up close at an animatronic, or robot, of the German writer Thomas Melle. The show proper – a lecture delivered by this robot – has finished and the audience has been invited on stage to study our automated actor. It feels like viewing time at the zoo and I can't shake the feeling that this robot is somehow going to wake up, reach out and grab me.

A figure sits alone on stage, dressed in comfy jumper and trousers, one leg crossed over the other. He slowly moves his hands and turns his head. But this sole performer in Uncanny Valley, by theatre company Rimini Protokoll, is not human. It is a lifelike animatronic model of the German writer Thomas Melle. The show's director, Stefan Kaegi, had seen animatronics used in museums, where he found there was not sufficient time for what he calls the "empathy mechanism" to kick in. But he wondered what would happen if the robot became a performer, "someone with whom we start to identify".

FIGURE 15-1 The major roles of EMYCIN: acquiring a knowledge base from the system designer, and interpreting that knowledge base to provide advice to a client.