Disentangling Textual and Acoustic Features of Neural Speech Representations

Neural speech models build deeply entangled internal representations, which capture a variety of features (e.g., fundamental frequency, loudness, syntactic category, or semantic content of a word) in a distributed encoding. This complexity makes it difficult to track the extent to which such representations rely on textual and acoustic information, or to suppress the encoding of acoustic features that may pose privacy risks (e.g., gender or speaker identity) in critical, real-world applications. In this paper, we build upon the Information Bottleneck principle to propose a disentanglement framework that separates complex speech representations into two distinct components: one encoding content (i.e., what can be transcribed as text) and the other encoding acoustic features relevant to a given downstream task. We apply and evaluate our framework to emotion recognition and speaker identification downstream tasks, quantifying the contribution of textual and acoustic features at each model layer. Additionally, we explore the application of our disentanglement framework as an attribution method to identify the most salient speech frame representations from both the textual and acoustic perspectives. The internal activation vectors of most modern deep learning systems, including Neural Speech Models (NSM) such as Wav2Vec2 (Baevski et al., 2020), HuBERT (Hsu et al., 2021), and Whisper (Radford et al., 2022), are highly entangled. This means that distinct input characteristics - such as fundamental frequency, loudness, syntactic category, or semantic features of a spoken word--are not separated into individual dimensions within the model's latent space - but are instead intertwined within the same ones. Entanglement is a major obstacle for our ability to interpret and to intervene; disentanglement, to the extent that it is possible and even if imperfect, is therefore often highly desirable. For instance, when state-of-the-art NSMs are used in critical situations, we may want to be able to guarantee that information about the speaker's identity, gender, or health characteristics are not used in downstream applications. However, the entangled nature of the NSM's internal representation makes it difficult to surgically suppress such acoustic information.