In the last step, we implement Midi synthesizers to generate realistic music.Audeoconverts video to audio smoothly and clearly withonlyafewsetupconstraints.Weevaluate Audeoonpianoperformancevideos collected from YouTube and obtain that their generated music is of reasonable audio quality andcanbesuccessfully recognized withhighprecision bypopular music identification software. The source code with examples is available in a Githubrepository3.

Summary and Contributions: This paper proposes a novel pipeline approach for improving piano music/audio generation from silent videos with a top-view of a pianist's fingers playing on a keyboard. Prior work [27] used an end-to-end approach to directly predict a symbolic piano performance from video using ResNets. This paper points out there's a lot of mismatch between the video and music/audio streams and hence the processing requires multiple stages of transformation. The proposed pipeline consists of three interpretable components / stages. Video2Roll consists of three stages.

Anyone who's been to a concert knows that something magical happens between the performers and their instruments. It transforms music from being just "notes on a page" to a satisfying experience. A University of Washington team wondered if artificial intelligence could recreate that delight using only visual cues--a silent, top-down video of someone playing the piano. The researchers used machine learning to create a system, called Audeo, that creates audio from silent piano performances. When the group tested the music Audeo created with music-recognition apps, such as SoundHound, the apps correctly identified the piece Audeo played about 86% of the time.