Cow goes moo: Artificial intelligence-based system associates images with sounds

A child can learn from a picture book to associate images with sounds, but building a computer vision system that can train itself isn't as simple. Using artificial intelligence techniques, however, researchers at Disney Research and ETH Zurich have designed a system that can automatically learn the association between images and the sounds they could plausibly make. Given a picture of a car, for instance, their system can automatically return the sound of a car engine. A system that knows the sound of a car, a splintering dish, or a slamming door might be used in a number of applications, such as adding sound effects to films, or giving audio feedback to people with visual disabilities, noted Jean-Charles Bazin, associate research scientist at Disney Research. To solve this challenging task, the research team leveraged data from collections of videos. "Videos with audio tracks provide us with a natural way to learn correlations between sounds and images," Bazin said.