Media
Automatic Music Transcription using Convolutional Neural Networks and Constant-Q transform
Telila, Yohannis, Cucinotta, Tommaso, Bacciu, Davide
Automatic music transcription (AMT) is the problem of analyzing an audio recording of a musical piece and detecting notes that are being played. AMT is a challenging problem, particularly when it comes to polyphonic music. The goal of AMT is to produce a score representation of a music piece, by analyzing a sound signal containing multiple notes played simultaneously. In this work, we design a processing pipeline that can transform classical piano audio files in .wav format into a music score representation. The features from the audio signals are extracted using the constant-Q transform, and the resulting coefficients are used as an input to the convolutional neural network (CNN) model.
VideoLLM Benchmarks and Evaluation: A Survey
The rapid development of Large Language Models (LLMs) has catalyzed significant advancements in video understanding technologies. This survey provides a comprehensive analysis of benchmarks and evaluation methodologies specifically designed or used for Video Large Language Models (VideoLLMs). We examine the current landscape of video understanding benchmarks, discussing their characteristics, evaluation protocols, and limitations. The paper analyzes various evaluation methodologies, including closed-set, open-set, and specialized evaluations for temporal and spatiotemporal understanding tasks. We highlight the performance trends of state-of-the-art VideoLLMs across these benchmarks and identify key challenges in current evaluation frameworks. Additionally, we propose future research directions to enhance benchmark design, evaluation metrics, and protocols, including the need for more diverse, multimodal, and interpretability-focused benchmarks. This survey aims to equip researchers with a structured understanding of how to effectively evaluate VideoLLMs and identify promising avenues for advancing the field of video understanding with large language models.
Facilitating Video Story Interaction with Multi-Agent Collaborative System
Zhang, Yiwen, Hao, Jianing, Wang, Zhan, Sheng, Hongling, Zeng, Wei
Video story interaction enables viewers to engage with and explore narrative content for personalized experiences. However, existing methods are limited to user selection, specially designed narratives, and lack customization. To address this, we propose an interactive system based on user intent. Our system uses a Vision Language Model (VLM) to enable machines to understand video stories, combining Retrieval-Augmented Generation (RAG) and a Multi-Agent System (MAS) to create evolving characters and scene experiences. It includes three stages: 1) Video story processing, utilizing VLM and prior knowledge to simulate human understanding of stories across three modalities. 2) Multi-space chat, creating growth-oriented characters through MAS interactions based on user queries and story stages. 3) Scene customization, expanding and visualizing various story scenes mentioned in dialogue. Applied to the Harry Potter series, our study shows the system effectively portrays emergent character social behavior and growth, enhancing the interactive experience in the video story world.
Beyond Misinformation: A Conceptual Framework for Studying AI Hallucinations in (Science) Communication
This paper proposes a conceptual framework for understanding AI hallucinations as a distinct form of misinformation. While misinformation scholarship has traditionally focused on human intent, generative AI systems now produce false yet plausible outputs absent of such intent. I argue that these AI hallucinations should not be treated merely as technical failures but as communication phenomena with social consequences. Drawing on a supply-and-demand model and the concept of distributed agency, the framework outlines how hallucinations differ from human-generated misinformation in production, perception, and institutional response. I conclude by outlining a research agenda for communication scholars to investigate the emergence, dissemination, and audience reception of hallucinated content, with attention to macro (institutional), meso (group), and micro (individual) levels. This work urges communication researchers to rethink the boundaries of misinformation theory in light of probabilistic, non-human actors increasingly embedded in knowledge production.
Netflix's new AI chatbot might lead you to your next binge-watch
We've all been there--sitting and staring at row after row of Netflix categories, trying to find something new to watch. I've browsed Netflix so much, I feel like I've scrolled for shows more than I've actually streamed them. Netflix has offered many solutions to try and help solve the video browsing blues, from the streaming's famous algorithm to the "play something" button. Now, Netflix has a new tool to help you find the perfect movie or show--and yes, AI has entered the chat, quite literally. Coming soon as a "small" opt-in only beta for the Netflix iOS app, Netflix's new AI chatbot will help you search for videos using "natural, conversational phrases" rather than just sifting through rows and rows of categories. For example, you'll be able to type "Something funny and upbeat" or "I want something scary, but not too scary" into the chat box, and Netflix's AI bot will serve up a list of suggestions, and with a comment like "Good vibes only: These comedies will leave you smiling, laughing or both."
Netflix overhauls its TV app with a fresh UI and responsive recommendations
Netflix is giving its TV user interface a major overhaul. Alongside a fresh, cleaner look, you'll see recommendations that adapt to your activity as Netflix tries to better gauge what you might be in the mood to watch. The company plans to roll out the update over the coming weeks and months. It seems that the aim here, as has long been a goal for Netflix, is to help you find something you want to watch faster. That way, you won't spend an eternity scrolling through the various options while struggling to figure out what to start streaming.
Evaluating the Impact of AI-Powered Audiovisual Personalization on Learner Emotion, Focus, and Learning Outcomes
Wang, George Xi, Deng, Jingying, Ali, Safinah
Independent learners often struggle with sustaining focus and emotional regulation in unstructured or distracting settings. Although some rely on ambient aids such as music, ASMR, or visual backgrounds to support concentration, these tools are rarely integrated into cohesive, learner-centered systems. Moreover, existing educational technologies focus primarily on content adaptation and feedback, overlooking the emotional and sensory context in which learning takes place. Large language models have demonstrated powerful multimodal capabilities including the ability to generate and adapt text, audio, and visual content. Educational research has yet to fully explore their potential in creating personalized audiovisual learning environments. To address this gap, we introduce an AI-powered system that uses LLMs to generate personalized multisensory study environments. Users select or generate customized visual themes (e.g., abstract vs. realistic, static vs. animated) and auditory elements (e.g., white noise, ambient ASMR, familiar vs. novel sounds) to create immersive settings aimed at reducing distraction and enhancing emotional stability. Our primary research question investigates how combinations of personalized audiovisual elements affect learner cognitive load and engagement. Using a mixed-methods design that incorporates biometric measures and performance outcomes, this study evaluates the effectiveness of LLM-driven sensory personalization. The findings aim to advance emotionally responsive educational technologies and extend the application of multimodal LLMs into the sensory dimension of self-directed learning.
The Inverse Drum Machine: Source Separation Through Joint Transcription and Analysis-by-Synthesis
Torres, Bernardo, Peeters, Geoffroy, Richard, Gael
--We present the Inverse Drum Machine (IDM), a novel approach to Drum Source Separation that leverages an analysis-by-synthesis framework combined with deep learning. Unlike recent supervised methods that require isolated stem recordings, our approach operates on drum mixtures with only transcription annotations. IDM integrates Automatic Drum Transcription and One-shot drum Sample Synthesis, jointly optimizing these tasks in an end-to-end manner . By convolving synthesized one-shot samples with estimated onsets, akin to a drum machine, we reconstruct the individual drum stems and train a Deep Neural Network on the reconstruction of the mixture. Experiments on the StemGMD dataset demonstrate that IDM achieves separation quality comparable to state-of-the-art supervised methods that require isolated stems data, while significantly outperforming matrix decomposition baselines. N Western popular music, the rhythmic foundation typically relies on percussion instruments from a standard drum kit comprising kick drum, snare drum, and hi-hat, while additional elements such as cymbals, tom-toms, and auxiliary percussions provide timbral complexity and rhythmic variation. Music producers and engineers often need to adjust individual drum instruments separately for remixing, rebalanc-ing, effects processing, or creating educational materials [1], [2]. Ideally, music production would utilize isolated recordings of each drum instrument (known as "stems"), allowing for precise control during mixing. However, these instruments are usually played simultaneously and by the same performer, resulting in recordings in which all elements are mixed into a single audio stream. Obtaining these separated stems during recording requires multiple microphones (leading to microphone bleeding) or asking musicians to play in unnatural conditions [3]. The need for tools that can extract individual drum stems from already mixed recordings has led to growing interest in Drum Source Separation (DSS). These solutions, however, are proprietary and still have limitations in separation quality and flexibility. DSS is challenging due to the acoustic properties of percussion sounds.
Modeling Musical Genre Trajectories through Pathlet Learning
Marey, Lilian, Laclau, Charlotte, Sguerra, Bruno, Viard, Tiphaine, Moussallam, Manuel
The increasing availability of user data on music streaming platforms opens up new possibilities for analyzing music consumption. However, understanding the evolution of user preferences remains a complex challenge, particularly as their musical tastes change over time. This paper uses the dictionary learning paradigm to model user trajectories across different musical genres. We define a new framework that captures recurring patterns in genre trajectories, called pathlets, enabling the creation of comprehensible trajectory embeddings. We show that pathlet learning reveals relevant listening patterns that can be analyzed both qualitatively and quantitatively. This work improves our understanding of users' interactions with music and opens up avenues of research into user behavior and fostering diversity in recommender systems. A dataset of 2000 user histories tagged by genre over 17 months, supplied by Deezer (a leading music streaming company), is also released with the code.
Mamba-Diffusion Model with Learnable Wavelet for Controllable Symbolic Music Generation
Zhang, Jincheng, Fazekas, György, Saitis, Charalampos
Mamba-Diffusion Model with Learnable Wavelet for Controllable Symbolic Music Generation 1 st Jincheng Zhang Centre for Digital Music Queen Mary University of London London, UK jincheng.zhang@qmul.ac.uk 2 nd Gy orgy Fazekas Centre for Digital Music Queen Mary University of London London, UK george.fazekas@qmul.ac.uk 3 rd Charalampos Saitis Centre for Digital Music Queen Mary University of London London, UK c.saitis@qmul.ac.uk Abstract --The recent surge in the popularity of diffusion models for image synthesis has attracted new attention to their potential for generation tasks in other domains. However, their applications to symbolic music generation remain largely under-explored because symbolic music is typically represented as sequences of discrete events and standard diffusion models are not well-suited for discrete data. We represent symbolic music as image-like pi-anorolls, facilitating the use of diffusion models for the generation of symbolic music. Moreover, this study introduces a novel diffusion model that incorporates our proposed Transformer-Mamba block and learnable wavelet transform. Classifier-free guidance is utilised to generate symbolic music with target chords. Our evaluation shows that our method achieves compelling results in terms of music quality and controllability, outperforming the strong baseline in pianoroll generation. Index T erms --symbolic music generation, deep learning, diffusion models, wavelet transform, Mamba I.