Researchers' AI system infers music from silent videos of musicians

In a study accepted to the upcoming 2020 European Conference on Computer Vision, MIT and MIT-IBM Watson AI Lab researchers describe an AI system -- Foley Music -- that can generate "plausible" music from silent videos of musicians playing instruments. They say it works on a variety of music performances and outperforms "several" existing systems in generating music that's pleasant to listen to. It's the researchers' belief an AI model that can infer music from body movements could serve as the foundation for a range of applications, from adding sound effects to videos automatically to creating immersive experiences in virtual reality. Studies from cognitive psychology suggest humans possess this skill -- even young children report that what they hear is influenced by the signals they receive from seeing a person speak, for example. Foley Music extracts 2D key points of people's bodies (25 total points) and fingers (21 points) from video frames as intermediate visual representations, which it uses to model body and hand movements.