Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates

The autoencoding task used to train NACs embeds a compressed latent representation of speech features into discrete sequences of codes, which are useful for training autoregressive transformers to complete downstream audio tasks such as Text-to-Speech (TTS) [2], Automatic Speech Recognition (ASR) [3] and Full Duplex Speech Modeling [4]; they can also be used as a domain-specific tokenized vocabulary that Large Language Models (LLMs) can be adapted to use for audio generation [5]. Conventionally, the most widely used NACs have utilized Residual V ector Quantization (RVQ) [6], where at each encoder output timestep, the encoded feature representation is quantized by a top-level'coarse' codebook, and additional codebooks quantize the residual error from each prior quantization operation. Although effective, RVQ presents training challenges, as propagating gradients to the codeword vectors to align them with the unquantized encoder outputs necessitates the use of auxiliary loss functions. This creates a delicate optimization problem that often leads to codebook collapse [7] where only a subset of codewords is used. Additionally, RVQ also requires a comparatively complicated downstream modeling setup, as the sequence length is expanded by the number of quantized residuals; mechanisms to model the hierarchical nature of RVQ codes commonly rely on two separate transformers that operate globally and locally [8].