Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls

ABSTRACT We propose Polyffusion, a diffusion model that generates polyphonic music scores by regarding music as imagelike piano roll representations. The model is capable of controllable music generation with two paradigms: internal control and external control. We show that by using tive modeling [14,15], symbolic music generation still suffers internal and external controls, Polyffusion unifies a from the lack of controllability and consistency at different wide range of music creation tasks, including melody generation time scales [16]. In our study, we experiment with given accompaniment, accompaniment generation the idea of using diffusion models to approach controllable given melody, arbitrary music segment inpainting, and music symbolic music generation. Experimental results Inspired by the high-quality and controllable image show that our model significantly outperforms existing generation that diffusion models have achieved in computer Transformer and sampling-based baselines, and using vision, we devise an image-like piano roll format as pre-trained disentangled representations as external conditions the input, and used a UNet-based diffusion model to stepwise yields more effective controls.