Goto

Collaborating Authors

 Government


Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge

arXiv.org Artificial Intelligence

Large language models (LLMs) can serve as judges that offer rapid and reliable assessments of other LLM outputs. However, models may systematically assign overly favorable ratings to their own outputs, a phenomenon known as self-bias, which can distort evaluations of true model performance. Previous studies often conflate genuine differences in model quality with bias or incorrectly assume that evaluations from LLMs and humans follow the same rating distributions. In this work, we present a statistical framework that explicitly formalizes assumptions under which self-bias can be identified and estimated. Our method models the difference in the scoring distribution that LLM-as-a-judge assigns to its own completions compared to other models, while accounting for the underlying quality of the completions provided by an independent, third-party judge (e.g., humans). Our method reliably isolates and quantifies self-bias, even when models vary in ability, ensuring that genuine performance differences are not mistaken for self-bias. We conduct an empirical analysis of self-bias on a large dataset (>5000 prompt-completion pairs) consisting of expert human annotations and judgments from nine different LLM judges. We find that some models, such as GPT-4o and Claude 3.5 Sonnet, systematically assign higher scores to their own outputs. These models also display family-bias; systematically assigning higher ratings to outputs produced by other models of the same family. Our findings highlight potential pitfalls of using LLM judges and offer practical guidance to mitigate biases when interpreting automated evaluations.


CISO: Species Distribution Modeling Conditioned on Incomplete Species Observations

arXiv.org Artificial Intelligence

Species distribution models (SDMs) are widely used to predict species' geographic distributions, serving as critical tools for ecological research and conservation planning. Typically, SDMs relate species occurrences to environmental variables representing abiotic factors, such as temperature, precipitation, and soil properties. However, species distributions are also strongly influenced by biotic interactions with other species, which are often overlooked. While some methods partially address this limitation by incorporating biotic interactions, they often assume symmetrical pairwise relationships between species and require consistent co-occurrence data. In practice, species observations are sparse, and the availability of information about the presence or absence of other species varies significantly across locations. To address these challenges, we propose CISO, a deep learning-based method for species distribution modeling Conditioned on Incomplete Species Observations. CISO enables predictions to be conditioned on a flexible number of species observations alongside environmental variables, accommodating the variability and incompleteness of available biotic data. We demonstrate our approach using three datasets representing different species groups: sPlotOpen for plants, SatBird for birds, and a new dataset, SatButterfly, for butterflies. Our results show that including partial biotic information improves predictive performance on spatially separate test sets. When conditioned on a subset of species within the same dataset, CISO outperforms alternative methods in predicting the distribution of the remaining species. Furthermore, we show that combining observations from multiple datasets can improve performance. CISO is a promising ecological tool, capable of incorporating incomplete biotic information and identifying potential interactions between species from disparate taxa.


Towards Integrated Alignment

arXiv.org Artificial Intelligence

As AI adoption expands across human society, the problem of aligning AI models to match human preferences remains a grand challenge. Currently, the AI alignment field is deeply divided between behavioral and representational approaches, resulting in narrowly aligned models that are more vulnerable to increasingly deceptive misalignment threats. In the face of this fragmentation, we propose an integrated vision for the future of the field. Drawing on related lessons from immunology and cybersecurity, we lay out a set of design principles for the development of Integrated Alignment frameworks that combine the complementary strengths of diverse alignment approaches through deep integration and adaptive coevolution. We highlight the importance of strategic diversity - deploying orthogonal alignment and misalignment detection approaches to avoid homogeneous pipelines that may be "doomed to success". We also recommend steps for greater unification of the AI alignment research field itself, through cross-collaboration, open model weights and shared community resources.


Operationalizing Serendipity: Multi-Agent AI Workflows for Enhanced Materials Characterization with Theory-in-the-Loop

arXiv.org Artificial Intelligence

The history of science is punctuated by serendipitous discoveries, where unexpected observations, rather than targeted hypotheses, opened new fields of inquiry. While modern autonomous laboratories excel at accelerating hypothesis testing, their optimization for efficiency risks overlooking these crucial, unplanned findings. To address this gap, we introduce SciLink, an open-source, multi-agent artificial intelligence framework designed to operationalize serendipity in materials research by creating a direct, automated link between experimental observation, novelty assessment, and theoretical simulations. The framework employs a hybrid AI strategy where specialized machine learning models perform quantitative analysis of experimental data, while large language models handle higher-level reasoning. These agents autonomously convert raw data from materials characterization techniques into falsifiable scientific claims, which are then quantitatively scored for novelty against the published literature. We demonstrate the framework's versatility across diverse research scenarios, showcasing its application to atomic-resolution and hyperspectral data, its capacity to integrate real-time human expert guidance, and its ability to close the research loop by proposing targeted follow-up experiments. By systematically analyzing all observations and contextualizing them, SciLink provides a practical framework for AI-driven materials research that not only enhances efficiency but also actively cultivates an environment ripe for serendipitous discoveries, thereby bridging the gap between automated experimentation and open-ended scientific exploration.


AquaChat++: LLM-Assisted Multi-ROV Inspection for Aquaculture Net Pens with Integrated Battery Management and Thruster Fault Tolerance

arXiv.org Artificial Intelligence

The global demand for aquaculture has surged over the past decade, driving the expansion of offshore fish farming systems such as net pens [1, 2]. These structures, while effective for large-scale fish production, are continuously exposed to harsh marine environments that can degrade structural integrity, compromise biosecurity, and increase the risk of fish escape or environmental contamination [3]. As a result, regular and reliable inspection of aquaculture net pens is critical to ensuring operational safety, productivity, and regulatory compliance [4]. Recent advances in underwater robotics, control systems, and computer vision have enabled significant progress in autonomous inspection [5, 6]. Remotely Operated Vehicles (ROVs), in particular, offer a practical platform for deploying sensing payloads such as cameras, sonars and performing close-range inspection in confined underwater environments [7]. However, most existing ROV-based systems operate in isolation, with limited autonomy and minimal adaptability to dynamic conditions such as power constraints, actuator degradation, and evolving mission demands [8, 9]. Moreover, mission planning and coordination typically require expert operators, limiting the scalability and responsiveness of these systems in real-world aquaculture operations [10, 11, 12]. To address these challenges, we propose AquaChat++, a novel framework that combines the reasoning capabilities of Large Language Models (LLMs) with multi-ROV coordination, battery-aware mission planning, and fault-tolerant control [13, 14]. Unlike traditional inspection pipelines that rely on fixed scripts or manual supervision, AquaChat++ enables natural language-driven task planning and dynamic allocation across multiple ROVs.


Historical Prediction Attention Mechanism based Trajectory Forecasting for Proactive Work Zone Safety in a Digital Twin Environment

arXiv.org Artificial Intelligence

Proactive safety systems aim to mitigate risks by anticipating potential conflicts between vehicles and enabling early intervention to prevent work zone-related crashes. This study presents an infrastructure-enabled proactive work zone safety warning system that leverages a Digital Twin environment, integrating real-time multi-sensor data, detailed High-Definition (HD) maps, and a historical prediction attention mechanism-based trajectory prediction model. Using a co-simulation environment that combines Simulation of Urban MObility (SUMO) and CAR Learning to Act (CARLA) simulators, along with Lanelet2 HD maps and the Historical Prediction Network (HPNet) model, we demonstrate effective trajectory prediction and early warning generation for vehicle interactions in freeway work zones. To evaluate the accuracy of predicted trajectories, we use two standard metrics: Joint Average Displacement Error (ADE) and Joint Final Displacement Error (FDE). Specifically, the infrastructure-enabled HPNet model demonstrates superior performance on the work-zone datasets generated from the co-simulation environment, achieving a minimum Joint FDE of 0.3228 meters and a minimum Joint ADE of 0.1327 meters, lower than the benchmarks on the Argoverse (minJointFDE: 1.0986 m, minJointADE: 0.7612 m) and Interaction (minJointFDE: 0.8231 m, minJointADE: 0.2548 m) datasets. In addition, our proactive safety warning generation application, utilizing vehicle bounding boxes and probabilistic conflict modeling, demonstrates its capability to issue alerts for potential vehicle conflicts.


Do Streetscapes Still Matter for Customer Ratings of Eating and Drinking Establishments in Car-Dependent Cities?

arXiv.org Artificial Intelligence

This study examines how indoor and outdoor aesthetics, streetscapes, and neighborhood features shape customer satisfaction at eating and dining establishments (EDEs) across different urban contexts, varying in car dependency, in Washington, DC. Using review photos and street view images, computer vision models quantified perceived safety and visual appeal. Ordinal logistic regression analyzed their effects on Yelp ratings. Findings reveal that both indoor and outdoor environments significantly impact EDE ratings, while streetscape quality's influence diminishes in car-dependent areas. The study highlights the need for context-sensitive planning that integrates indoor and outdoor factors to enhance customer experiences in diverse settings.


Recommendation with Generative Models

arXiv.org Artificial Intelligence

Generative models are a class of AI models capable of creating new instances of data by learning and sampling from their statistical distributions. In recent years, these models have gained prominence in machine learning due to the development of approaches such as generative adversarial networks (GANs), variational autoencoders (VAEs), and transformer-based architectures such as GPT. These models have applications across various domains, such as image generation, text synthesis, and music composition. In recommender systems, generative models, referred to as Gen-RecSys, improve the accuracy and diversity of recommendations by generating structured outputs, text-based interactions, and multimedia content. By leveraging these capabilities, Gen-RecSys can produce more personalized, engaging, and dynamic user experiences, expanding the role of AI in eCommerce, media, and beyond. Our book goes beyond existing literature by offering a comprehensive understanding of generative models and their applications, with a special focus on deep generative models (DGMs) and their classification. We introduce a taxonomy that categorizes DGMs into three types: ID-driven models, large language models (LLMs), and multimodal models. Each category addresses unique technical and architectural advancements within its respective research area. This taxonomy allows researchers to easily navigate developments in Gen-RecSys across domains such as conversational AI and multimodal content generation. Additionally, we examine the impact and potential risks of generative models, emphasizing the importance of robust evaluation frameworks.


WIRED Roundup: Unpacking OpenAI's Government Partnership

WIRED

On today's episode, our host Zoรซ Schiffer is joined by WIRED's senior politics writer Jake Lahut to run through five of the most important stories we published this week--from how bitcoin miners have been racing this year to beat the tariffs, to how AI was used to find a missing hiker in the Italian Alps. Then, Zoรซ and Jake discuss the details around OpenAI's latest partnership with the federal government. Mentioned in this episode: OpenAI Announces Massive US Government Partnership by Zoรซ Schiffer and Will Knight Trumpworld Knows Epstein Is a Problem. But They Can't Solve It by Jake Lahut Charter Planes and Bidding Wars: How Bitcoin Miners Raced to Beat Trump's Tariffs by Joel Khalili Google Will Use AI to Guess People's Ages Based on Search History by Dell Cameron US Coast Guard Report on Titan Submersible Implosion Singles Out OceanGate CEO Stockton Rush by Mark Harris A Hiker Was Missing for Nearly a Year--Until an AI System Recognized His Helmet by Marta Abbร  Get tickets to our live show, happening on September 9th, here. Write to us at uncannyvalley@wired.com.


Trump lashes out at Crockett, renews call for cognitive test

FOX News

President Donald Trump has renewed his call for Rep. Jasmine Crockett, D-Texas, to undergo a cognitive test. "'Congresswoman' Jasmine Crockett is a Low (Very!!!) I.Q. Individual, much in the mold of the AOC Plus Three Gang of Country Destroying Morons - Only slightly dumber," Trump wrote on TRUTH Social on Monday. "Each of these political hacks should be forced to take a Cognitive Exam, much like the one I recently took while getting my'physical' at our GREAT Washington, D.C., Military Hospital (WR!)," Trump said. "As the doctors said, 'President Trump ACED it, something that is rarely seen!' These Radical Left Lunatics would all fail this test in a spectacular show of stupidity and incompetence. President Donald Trump demanded Texas Democrat Jasmine Crockett take a cognitive test as their public feud escalates. Trump previously said Rep. Alexandria Ocasio-Cortez, D-N.Y., should take a cognitive test in June when the progressive "Squad" leader demanded his impeachment over the U.S. strikes on Iranian nuclear facilities. Meanwhile, as the White House pushes Republican states to redistrict mid-cycle ahead of the 2026 midterm elections, Crockett has accused Trump of pushing a "white supremacy agenda" and "diluting the voices of people of color." The Trump administration asserts that Democratic states have engaged in "gerrymandering" for years and encouraged illegal immigration to boost their congressional influence. In Texas, Democratic state lawmakers fled the state in an effort to stop the vote on a GOP redistricting plan that likely would have resulted in Republicans picking up five House seats. Crockett has accused Trump of hurling the low IQ insult as a racially-coded tactic to insult "people of color," including "The Breakfast Club" host Charlamagne tha God. Rep. Jasmine Crockett, D-Texas, joins Texas state Democrats for a press conference on Aug. 4, 2025 in Warrenville, Illinois. "Newsflash, Wannabe Dictator: I don't care how many times you shake the Etch A Sketch trying to redraw these lines," Crockett wrote on X last week. I'll be back, still on your behind every step of the way. We've already been over this. I've got the degrees, the credentials, and the receipts. Despite the president describing her as having a low IQ, Crockett said Trump has the "most incompetent Cabinet in the history of this country," referring to the Signal-gate scandal earlier this year. Crockett has also dubbed Trump a "Temu dictator." At a progressive rally in Phoenix, Arizona, earlier this month, the congresswoman said on stage, "Donald Trump is a piece of sh--." "This is a person who has a problem with people of color.