Learning to Play Piano in the Real World

Abstract--Towards the grand challenge of achieving humanlevel manipulation in robots, playing piano is a compelling testbed that requires strategic, precise, and flowing movements. Over the years, several works demonstrated hand-designed controllers on real world piano playing, while other works evaluated robot learning approaches on simulated piano scenarios. In this paper, we develop the first piano playing robotic system that makes use of learning approaches while also being deployed on a real world dexterous robot. Specifically, we make use of Sim2Real to train a policy in simulation using reinforcement learning before deploying the learned policy on a real world dexterous robot. In our experiments, we thoroughly evaluate the interplay between domain randomization and the accuracy of the dynamics model used in simulation. Moreover, we evaluate the robot's performance across multiple songs with varying complexity to study the generalization of our learned policy. Experimental results show that the robot can learn Playing the piano requires humans to master contact-rich to play several simple pieces successfully, after training exclusively hand movements dictated by the timing and tone they intend in simulation. This mastery is not learned quickly but through extensive practice, which requires humans to control their actions based on the haptic and auditory feedback received the natural movements of human hands. This makes it an ideal with each key pressed on the piano. In addition, human hands scenario for exploring Sim2Real transfer, where the objective are an extraordinary research subject due to their unmatched is to train an agent in simulation capable of performing in the dexterity, precision, and adaptability.