Google has taken the idea of Rock Band and Guitar Hero and pushed it one step further, creating an intelligent controller that lets you improvise on the piano and makes it sound like you actually know what you're doing, no matter how unskilled you are. The controller is called Piano Genie, and it comes from Google's Magenta research project. Powered by a neural network trained on classical piano music, Piano Genie translates what you tap out on eight buttons into music that uses all 88 piano keys. "I really wanted to design a tool that we could give to someone who doesn't know how to play, and they'd be able to create music with some kind of intention," Google Magenta intern Chris Donahue, told The Verge. "When you're playing it, it's this really awesome experience where, occasionally, it will feel like it's sort of reading your mind and play the exact note you're intending to. And then other times, it will completely disobey you but still do something reasonable."

We present Piano Genie, an intelligent controller which allows non-musicians to improvise on the piano. With Piano Genie, a user performs on a simple interface with eight buttons, and their performance is decoded into the space of plausible piano music in real time. To learn a suitable mapping procedure for this problem, we train recurrent neural network autoencoders with discrete bottlenecks: an encoder learns an appropriate sequence of buttons corresponding to a piano piece, and a decoder learns to map this sequence back to the original piece. During performance, we substitute a user's input for the encoder output, and play the decoder's prediction each time the user presses a button. To improve the interpretability of Piano Genie's performance mechanics, we impose musically-salient constraints over the encoder's outputs.