Generative Disco: Text-to-Video Generation for Music Visualization
Liu, Vivian, Long, Tao, Raw, Nathan, Chilton, Lydia
–arXiv.org Artificial Intelligence
Visuals can enhance our experience of music, owing to the way they can amplify the emotions and messages conveyed within it. However, creating music visualization is a complex, time-consuming, and resource-intensive process. We introduce Generative Disco, a generative AI system that helps generate music visualizations with large language models and text-to-video generation. The system helps users visualize music in intervals by finding prompts to describe the images that intervals start and end on and interpolating between them to the beat of the music. We introduce design patterns for improving these generated videos: transitions, which express shifts in color, time, subject, or style, and holds, which help focus the video on subjects. A study with professionals showed that transitions and holds were a highly expressive framework that enabled them to build coherent visual narratives. We conclude on the generalizability of these patterns and the potential of generated video for creative professionals.
arXiv.org Artificial Intelligence
Sep-28-2023
- Country:
- Asia
- Japan > Honshū
- Kansai > Osaka Prefecture
- Osaka (0.04)
- Kantō
- Kanagawa Prefecture > Yokohama (0.04)
- Tokyo Metropolis Prefecture > Tokyo (0.14)
- Kansai > Osaka Prefecture
- South Korea > Seoul
- Seoul (0.04)
- Japan > Honshū
- Europe
- France > Auvergne-Rhône-Alpes
- Germany > Hamburg (0.04)
- Italy > Veneto
- Venice (0.04)
- Netherlands > North Holland
- Amsterdam (0.04)
- United Kingdom
- England > Cambridgeshire
- Cambridge (0.04)
- Scotland
- City of Glasgow > Glasgow (0.04)
- Fife > St. Andrews (0.04)
- England > Cambridgeshire
- North America > United States
- California
- Colorado (0.04)
- Hawaii > Honolulu County
- Honolulu (0.04)
- Louisiana > Orleans Parish
- New Orleans (0.04)
- Massachusetts
- Middlesex County > Cambridge (0.04)
- Suffolk County > Boston (0.04)
- New York > New York County
- New York City (0.07)
- North Carolina > Mecklenburg County
- Charlotte (0.04)
- Texas > Travis County
- Austin (0.04)
- Oceania > Australia
- New South Wales > Sydney (0.04)
- South America > Chile
- Asia
- Genre:
- Questionnaire & Opinion Survey (0.93)
- Research Report > New Finding (0.46)
- Industry:
- Leisure & Entertainment (1.00)
- Media
- Technology: