Universal Semantic Disentangled Privacy-preserving Speech Representation Learning

The use of audio recordings of human speech to train LLMs poses privacy concerns due to these models' potential to generate outputs that closely resemble artifacts in the training data. In this study, we propose a speaker privacy-preserving representation learning method through the Universal Speech Codec (USC), a computationally efficient encoder-decoder model that disentangles speech into: ( i) privacy-preserving semantically rich representations, capturing content and speech paralinguistics, and ( ii) residual acoustic and speaker representations that enables high-fidelity reconstruction. Extensive evaluations presented show that USC's semantic representation preserves content, prosody, and sentiment, while removing potentially identifiable speaker attributes. Combining both representations, USC achieves state-of-the-art speech reconstruction. Additionally, we introduce an evaluation methodology for measuring privacy-preserving properties, aligning with perceptual tests. We compare USC against other codecs in the literature and demonstrate its effectiveness on privacy-preserving representation learning, illustrating the trade-offs of speaker anonymization, paralinguistics retention and content preservation in the learned semantic representations. Audio samples are shared in https://www.amazon.science/usc-samples . Latest foundational Generative AI (GenAI) revolve around multimodality (Achiam et al., 2023; Anil et al., 2023; Dubey et al., 2024). The extraordinary capabilities of Large Language Models (LLMs) as multimodal learning machines have ushered in a new paradigm for what GenAI can offer to our world (Team, 2025). These foundational LLMs are data-hungry, requiring massive amounts of multimodal training data. Speech and audio are essential modalities for many applications, and mul-timodal models require exposure to them during their training process (Borsos et al., 2023). Speech is a form of individual information (Nautsch et al., 2019), and the development of new foundational speech-aware models demands access to massive amounts of speech data to fully unlock their potential. The research community has collected and curated public data over the past decades, which has been used for specialized speech models (Łajszczak et al., 2024). However, in the realm of Responsible AI, every individual and organization must make proper use of individuals' data when training foundational models, regardless of its public availability. Hence, privacy-preserving methods must be developed to advance foundational speech research while safeguarding individual privacy. Foundational LLMs trained on language modeling tasks model the likelihood of generating coherent text sequences from a distribution of discrete tokens (Touvron et al., 2023). This allows them to produce expressive and varied responses during generation.