Combining Visual and Acoustic Speech Signals with a Neural Network Improves Intelligibility

Previous attempts at using these visual speech signals to improve automatic speech recognition systems havecombined the acoustic and visual speech information at a symbolic level using heuristic rules. In this paper, we demonstrate an alternative approach to fusing the visual and acoustic speech information by training feedforward neural networks to map the visual signal onto the corresponding short-term spectral amplitude envelope (STSAE) of the acoustic signal. This information can be directly combined with the degraded acoustic STSAE. Significant improvementsare demonstrated in vowel recognition from noise-degraded acoustic signals. These results are compared to the performance of humans, as well as other pattern matching and estimation algorithms. 1 INTRODUCTION Current automatic speech recognition systems rely almost exclusively on the acoustic speechsignal, and as a consequence, these systems often perform poorly in noisy Combining Visual and Acoustic Speech Signals 233 environments.