The "cocktail party effect" describes humans' ability to hold a conversation in a noisy environment by listening to what their conversation partner is saying while filtering out other chatter, music, ambient noises, etc. We do it naturally but the problem has been widely studied in machine learning, where the development of environmental sound recognition and source separation techniques that can tune into a single sound and filter out all others is a research focus. MIT CSAIL researchers recently introduced their PixelPlayer system, which has learned to identify objects that produce sound in videos. The system uses deep learning and was trained by binge-watching 60 hours of musical performances to identify the natural synchronization of visual and audio information. The team trained deep neural networks to concentrate on images and audio and identify pixel-level image locations for sound sources in the videos.

We listen to music with our ears, but also our eyes, watching with appreciation as the pianist's fingers fly over the keys and the violinist's bow rocks across the ridge of strings. When the ear fails to tell two instruments apart, the eye often pitches in by matching each musician's movements to the beat of each part. A new artificial intelligence tool developed by the MIT-IBM Watson AI Lab leverages the virtual eyes and ears of a computer to separate similar sounds that are tricky even for humans to differentiate. The tool improves on earlier iterations by matching the movements of individual musicians, via their skeletal keypoints, to the tempo of individual parts, allowing listeners to isolate a single flute or violin among multiple flutes or violins. Potential applications for the work range from sound mixing, and turning up the volume of an instrument in a recording, to reducing the confusion that leads people to talk over one another on a video-conference calls.

If you've ever played a YouTube video for what it seems like the thousandth time to listen to your instrument's part of a composition, you'll love MIT's new AI. PixelPlayer, which hails from the institution's Computer Science and Artificial Intelligence Laboratory (CSAIL), can recognize instruments in a video, identify specific ones at pixel level and isolate the sounds they produce. If there are several instruments playing in a video, for instance, PixelPlayer will allow you to pick the one you want to listen to -- it will play the sounds coming out of that instrument the loudest and will lower the volume or everything else. CSAIL trained PixelPlayer by using a self-supervised deep learning technique and feeding it over 60 hours of videos to learn from. It's still far from perfect, though: it can only identify the sounds of 20 instruments at the moment and is still having trouble telling similar ones apart.