Towards Audio Codec-based Speech Separation

Recent improvements in neural audio codec (NAC) models have generated interest in adopting pre-trained codecs for a variety of speech processing applications to take advantage of the efficiencies gained from high compression, but these have yet been applied to the speech separation (SS) task. SS can benefit from high compression because the compute required for traditional SS models makes them impractical for many edge computing use cases. However, SS is a waveform-masking task where compression tends to introduce distortions that severely impact performance. Here we propose a novel task of Audio Codecbased SS, where SS is performed within the embedding space of a NAC, and propose a new model, Codecformer, to address Figure 1: (Top) Overview of Codecformer performing speech this task. At inference, Codecformer achieves a 52x reduction separation within the embedding space of an audio codec. The in MAC while producing separation performance comparable decoding can happen either locally or in the cloud.