Music foundation models possess impressive music generation capabilities. When people compose music, they may infuse their understanding of music into their work, by using notes and intervals to craft melodies, chords to build progressions, and tempo to create a rhythmic feel. To what extent is this true of music generation models? More specifically, are fundamental Western music theory concepts observable within the "inner workings" of these models? Recent work proposed leveraging latent audio representations from music generation models towards music information retrieval tasks (e.g. genre classification, emotion recognition), which suggests that high-level musical characteristics are encoded within these models. However, probing individual music theory concepts (e.g. tempo, pitch class, chord quality) remains under-explored. Thus, we introduce SynTheory, a synthetic MIDI and audio music theory dataset, consisting of tempos, time signatures, notes, intervals, scales, chords, and chord progressions concepts. We then propose a framework to probe for these music theory concepts in music foundation models (Jukebox and MusicGen) and assess how strongly they encode these concepts within their internal representations. Our findings suggest that music theory concepts are discernible within foundation models and that the degree to which they are detectable varies by model size and layer.

Large language models (LLMs) have become increasingly pivotal across various domains, especially in handling complex data types. This includes structured data processing, as exemplified by ChartQA and ChatGPT-Ada, and multimodal unstructured data processing as seen in Visual Question Answering (VQA). These areas have attracted significant attention from both industry and academia. Despite this, there remains a lack of unified evaluation methodologies for these diverse data handling scenarios. In response, we introduce BabelBench, an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. BabelBench incorporates a dataset comprising 247 meticulously curated problems that challenge the models with tasks in perception, commonsense reasoning, logical reasoning, and so on. Besides the basic capabilities of multimodal understanding, structured data processing as well as code generation, these tasks demand advanced capabilities in exploration, planning, reasoning and debugging. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement. The insights derived from our comprehensive analysis offer valuable guidance for future research within the community. The benchmark data can be found at https://github.com/FFD8FFE/babelbench.

Volumetric modeling and neural radiance field representations have revolutionized 3D face capture and photorealistic novel view synthesis. However, these methods often require hundreds of multi-view input images and are thus inapplicable to cases with less than a handful of inputs. We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling from as few as three input views captured in the wild. Our key insight is that an implicit prior trained on synthetic data alone can generalize to extremely challenging real-world identities and expressions and render novel views with fine idiosyncratic details like wrinkles and eyelashes. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions, hair, clothing, and other assets. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject. On average, the fine-tuning requires only three inputs to cross the synthetic-to-real domain gap. The resulting personalized 3D model reconstructs strong idiosyncratic facial expressions and outperforms the state-of-the-art in high-quality novel view synthesis of faces from sparse inputs in terms of perceptual and photo-metric quality.

Recommender systems are designed to suggest items based on user preferences, helping users navigate the vast amount of information available on the internet. Given the overwhelming content, outlier detection has emerged as a key research area in recommender systems. It involves identifying unusual or suspicious patterns in user behavior. However, existing studies in this field face several challenges, including the limited universality of algorithms, difficulties in selecting users, and a lack of optimization. In this paper, we propose an approach that addresses these challenges by employing various clustering algorithms. Specifically, we utilize a user-user matrix-based clustering technique to detect outliers. By constructing a user-user matrix, we can identify suspicious users in the system. Both local and global outliers are detected to ensure comprehensive analysis. Our experimental results demonstrate that this approach significantly improves the accuracy of outlier detection in recommender systems.

Recent advances in large language models (LLMs) have significantly impacted the domain of multi-hop question answering (MHQA), where systems are required to aggregate information and infer answers from disparate pieces of text. However, the autoregressive nature of LLMs inherently poses a challenge as errors may accumulate if mistakes are made in the intermediate reasoning steps. This paper introduces Monte-Carlo tree search for Zero-shot multi-hop Question Answering (MZQA), a framework based on Monte-Carlo tree search (MCTS) to identify optimal reasoning paths in MHQA tasks, mitigating the error propagation from sequential reasoning processes. Unlike previous works, we propose a zero-shot prompting method, which relies solely on instructions without the support of hand-crafted few-shot examples that typically require domain expertise. We also introduce a behavioral cloning approach (MZQA-BC) trained on self-generated MCTS inference trajectories, achieving an over 10-fold increase in reasoning speed with bare compromise in performance. The efficacy of our method is validated on standard benchmarks such as HotpotQA, 2WikiMultihopQA, and MuSiQue, demonstrating that it outperforms existing frameworks.

So far, when AI companies have trained on YouTube's invaluable stash of videos, captions, and other content, they've done so without permission. An AI-focused content licensing startup called Calliope Networks is hoping to change that with its new "License to Scrape," a program aimed directly at YouTube stars. "There's obvious demand from AI companies to scrape YouTube content. We see that by their actions. So what we're trying to do is to create a tool that makes it legal and simple for them," says Calliope Networks CEO Dave Davis.

Experts are warning foragers to avoid using Google Images to identify mushrooms after the search engine is delivering misleading AI-generated results. Searches for a number of common edible mushrooms return wildly inaccurate images as the top result, despite these images being flagged as AI-generated. Foraging experts warn this could lead to dangerous, if not deadly, errors for foragers trying to identify safe mushrooms to eat. Professor David Hawksworth, a mycologist from the University of Southampton, told MailOnline: 'This is potentially extremely dangerous.' However, experts routinely warn that it isn't safe to pick up and eat mushrooms that we find on the ground - even if we think we can tell a safe species apart from a dangerous one.

Ayad Akhtar's brilliant new play, McNeal, currently at the Lincoln Center Theater, is transfixing in part because it tracks without flinching the disintegration of a celebrated writer, and in part because Akhtar goes to a place that few writers have visited so effectively--the very near future, in which large language models threaten to undo our self-satisfied understanding of creativity, plagiarism, and originality. And also because Robert Downey Jr., performing onstage for the first time in more than 40 years, perfectly embodies the genius and brokenness of the title character. Check out more from this issue and find your next story to read. I've been in conversation for quite some time with Akhtar, whose play Disgraced won the Pulitzer Prize in 2013, about artificial generative intelligence and its impact on cognition and creation. He's one of the few writers I know whose position on AI can't be reduced to the (understandable) plea For God's sake, stop threatening my existence! In McNeal, he not only suggests that LLMs might be nondestructive utilities for human writers, but also deployed LLMs as he wrote (he's used many of them, ChatGPT, Claude, and Gemini included). To my chagrin and astonishment, they seem to have helped him make an even better play. As you will see in our conversation, he doesn't believe that this should be controversial. In early September, Akhtar, Downey, Bartlett Sher--the Tony Award winner who directed McNeal--and I met at Downey's home in New York for what turned out to be an amusing, occasionally frenetic, and sometimes even borderline profound discussion of the play, its origins, the flummoxing issues it raises, and, yes, Avengers: Age of Ultron. We were joined intermittently by Susan Downey, Robert's wife (and producing partner), and the person who believed that Akhtar's play would tempt her husband to return to the stage. The conversation that follows is a condensed and edited version of our sprawling discussion, but I think it captures something about art and AI, and it certainly captures the exceptional qualities of three people, writer, director, and actor, who are operating at the pinnacle of their trade, without fear--perhaps without enough fear--of what is inescapably coming.

Backstage at one of the largest artificial intelligence conferences in the world, Gov. Gavin Newsom listened to two leaders in the field debate opposite views of a high-profile bill on his desk to protect Californians from the technology. "Honestly, I take advantage of opportunities like this," Newsom said recounting the exchange later during an interview at the Salesforce conference in San Francisco in mid-September. "I just watched them, and I was like, 'Here we go. Should I sign it, or should I not?' Then'absolutely,' 'absolutely not' and back and forth." The scene offered a peek into Newsom's deliberations on regulating the tech industry, including an explosion of AI companies, and the forces seeking to influence him during bill-signing season at the state Capitol.

Despite demonstrating superior performance across a variety of linguistic tasks, pre-trained large language models (LMs) often require fine-tuning on specific datasets to effectively address different downstream tasks. However, fine-tuning these LMs for downstream tasks necessitates collecting data from individuals, which raises significant privacy concerns. Federated learning (FL) has emerged as the de facto solution, enabling collaborative model training without sharing raw data. While promising, federated fine-tuning of large LMs faces significant challenges, including restricted access to model parameters and high computation, communication, and memory overhead. To address these challenges, this paper introduces \textbf{Fed}erated \textbf{P}roxy-\textbf{T}uning (FedPT), a novel framework for federated fine-tuning of black-box large LMs, requiring access only to their predictions over the output vocabulary instead of their parameters. Specifically, devices in FedPT first collaboratively tune a smaller LM, and then the server combines the knowledge learned by the tuned small LM with the knowledge learned by the larger pre-trained LM to construct a large proxy-tuned LM that can reach the performance of directly tuned large LMs. The experimental results demonstrate that FedPT can significantly reduce computation, communication, and memory overhead while maintaining competitive performance compared to directly federated fine-tuning of large LMs. FedPT offers a promising solution for efficient, privacy-preserving fine-tuning of large LMs on resource-constrained devices, broadening the accessibility and applicability of state-of-the-art large LMs.