Goto

Collaborating Authors

 Clustering




Benchmarking Prosody Encoding in Discrete Speech Tokens

arXiv.org Artificial Intelligence

--Recently, discrete tokens derived from self-supervised learning (SSL) models via k-means clustering have been actively studied as pseudo-text in speech language models and as efficient intermediate representations for various tasks. However, these discrete tokens are typically learned in advance, separately from the training of language models or downstream tasks. As a result, choices related to discretization, such as the SSL model used or the number of clusters, must be made heuristically. In particular, speech language models are expected to understand and generate responses that reflect not only the semantic content but also prosodic features. Y et, there has been limited research on the ability of discrete tokens to capture prosodic information. T o address this gap, this study conducts a comprehensive analysis focusing on prosodic encoding based on their sensitivity to the artificially modified prosody, aiming to provide practical guidelines for designing discrete tokens. Self-supervised learning (SSL) models [1]-[5] have demonstrated strong performance across various speech processing tasks and have been widely adopted in the field. Recently, there has been growing interest in discretizing their outputs for more effective utilization [6]. By discretizing speech, these tokens can be seen as "pseudo-text" and now (large) language models can process speech directly without transcribing it into text [7]-[11].






Supplement to " Efficient Clustering for Stretched Mixtures: Landscape and Optimality "

Neural Information Processing Systems

K-means (DisKmeans) in Y e et al. (2008); (ii) a discriminative clustering formulation described in Bach and Harchaoui (2008); Flammarion et al. (2017); (iii) Model-based clustering (Mclust) in Fraley and Raftery (1999); (iv) Projection Pursuit (PP) in Peรฑa and Prieto (2001); (v) Adaptive LDA-guided K-means Clustering in Ding and Li (2007); and (vi) Minimum Density Hyperplane As suggested by Y e et al. (2008), the regularization parameter To resolve this issue, they provide an automatic tuning framework. Here we provide a comparison between CURE and DisKmeans. Density Hyperplane (MDH) in Pavlidis et al. (2016) are implemented using open-source R packages The discriminative clustering method appeared in Bach and Harchaoui (2008); Flammarion et al. The iterative algorithm is terminated when y, the predicted label, no longer changes. To illustrate how the general CURE in Section 2.3 works, we consider the clustering problem with When a is sufficiently large and b 2a, f has the following properties: 1. f B.2 Step 2: landscape analysis of the population loss To kick off the landscape analysis we investigate the population version of ห† L See Appendix E for a proof.