Government
RAVine: Reality-Aligned Evaluation for Agentic Search
Xu, Yilong, Long, Xiang, Zheng, Zhi, Gao, Jinhua
Agentic search, as a more autonomous and adaptive paradigm of retrieval augmentation, is driving the evolution of intelligent search systems. However, existing evaluation frameworks fail to align well with the goals of agentic search. First, the complex queries commonly used in current benchmarks often deviate from realistic user search scenarios. Second, prior approaches tend to introduce noise when extracting ground truth for end-to-end evaluations, leading to distorted assessments at a fine-grained level. Third, most current frameworks focus solely on the quality of final answers, neglecting the evaluation of the iterative process inherent to agentic search. To address these limitations, we propose RAVine -- a Reality-Aligned eValuation framework for agentic LLMs with search. RAVine targets multi-point queries and long-form answers that better reflect user intents, and introduces an attributable ground truth construction strategy to enhance the accuracy of fine-grained evaluation. Moreover, RAVine examines model's interaction with search tools throughout the iterative process, and accounts for factors of efficiency. We benchmark a series of models using RAVine and derive several insights, which we hope will contribute to advancing the development of agentic search systems. The code and datasets are available at https://github.com/SwordFaith/RAVine.
Optimizing Start Locations in Ergodic Search for Disaster Response
Rao, Ananya, Hargis, Alyssa, Wettergreen, David, Choset, Howie
In disaster response scenarios, deploying robotic teams effectively is crucial for improving situational awareness and enhancing search and rescue operations. The use of robots in search and rescue has been studied but the question of where to start robot deployments has not been addressed. This work addresses the problem of optimally selecting starting locations for robots with heterogeneous capabilities by formulating a joint optimization problem. To determine start locations, this work adds a constraint to the ergodic optimization framework whose minimum assigns robots to start locations. This becomes a little more challenging when the robots are heterogeneous (equipped with different sensing and motion modalities) because not all robots start at the same location, and a more complex adaptation of the aforementioned constraint is applied. Our method assumes access to potential starting locations, which can be obtained from expert knowledge or aerial imagery. We experimentally evaluate the efficacy of our joint optimization approach by comparing it to baseline methods that use fixed starting locations for all robots. Our experimental results show significant gains in coverage performance, with average improvements of 35.98% on synthetic data and 31.91% on real-world data for homogeneous and heterogeneous teams, in terms of the ergodic metric.
Framing Political Bias in Multilingual LLMs Across Pakistani Languages
Nadeem, Afrozah, Dras, Mark, Naseem, Usman
Large Language Models (LLMs) increasingly shape public discourse, yet most evaluations of political and economic bias have focused on high-resource, Western languages and contexts. This leaves critical blind spots in low-resource, multilingual regions such as Pakistan, where linguistic identity is closely tied to political, religious, and regional ideologies. We present a systematic evaluation of political bias in 13 state-of-the-art LLMs across five Pakistani languages: Urdu, Punjabi, Sindhi, Pashto, and Balochi. Our framework integrates a culturally adapted Political Compass Test (PCT) with multi-level framing analysis, capturing both ideological stance (economic/social axes) and stylistic framing (content, tone, emphasis). Prompts are aligned with 11 socio-political themes specific to the Pakistani context. Results show that while LLMs predominantly reflect liberal-left orientations consistent with Western training data, they exhibit more authoritarian framing in regional languages, highlighting language-conditioned ideological modulation. We also identify consistent model-specific bias patterns across languages. These findings show the need for culturally grounded, multilingual bias auditing frameworks in global NLP.
Intersectional Divergence: Measuring Fairness in Regression
Germino, Joe, Moniz, Nuno, Chawla, Nitesh V.
Fairness in machine learning research is commonly framed in the context of classification tasks, leaving critical gaps in regression. In this paper, we propose a novel approach to measure intersectional fairness in regression tasks, going beyond the focus on single protected attributes from existing work to consider combinations of all protected attributes. Furthermore, we contend that it is insufficient to measure the average error of groups without regard for imbalanced domain preferences. Accordingly, we propose Intersectional Divergence (ID) as the first fairness measure for regression tasks that 1) describes fair model behavior across multiple protected attributes and 2) differentiates the impact of predictions in target ranges most relevant to users. We extend our proposal demonstrating how ID can be adapted into a loss function, IDLoss, that satisfies convergence guarantees and has piecewise smooth properties that enable practical optimization. Through an extensive experimental evaluation, we demonstrate how ID allows unique insights into model behavior and fairness, and how incorporating IDLoss into optimization can considerably improve single-attribute and intersectional model fairness while maintaining a competitive balance in predictive performance.
Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Sรผhr, Tom, Dorner, Florian E., Salaudeen, Olawale, Kelava, Augustin, Samadi, Samira
Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intelligence and personality. While these results are often interpreted as strong evidence of human-like characteristics in LLMs, this paper argues that such interpretations constitute an ontological error. Human psychological and educational tests are theory-driven measurement instruments, calibrated to a specific human population. Applying these tests to non-human subjects without empirical validation, risks mischaracterizing what is being measured. Furthermore, a growing trend frames AI performance on benchmarks as measurements of traits such as ``intelligence'', despite known issues with validity, data contamination, cultural bias and sensitivity to superficial prompt changes. We argue that interpreting benchmark performance as measurements of human-like traits, lacks sufficient theoretical and empirical justification. This leads to our position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead. We call for the development of principled, AI-specific evaluation frameworks tailored to AI systems. Such frameworks might build on existing frameworks for constructing and validating psychometrics tests, or could be created entirely from scratch to fit the unique context of AI.
Apple quietens Wall Street's fears of China struggles and slow AI progress
Apple has been under pressure this year. It's playing catch-up to its fellow tech giants on artificial intelligence, it's seen its stock fall by double digits since the year began, it closed a store in China for the first time ever this week, and looming US tariffs on Beijing threaten its supply chain. On Thursday, the company released its third-quarter earnings of the fiscal year as investors scrutinize how the iPhone maker might turn things around. Despite the gloomy outlook, the company is still worth more than 3tn, and it beat Wall Street's expectations for profit and revenue this quarter. Apple reported a massive 10% year-over-year increase in revenue to 94.04bn, and 1.57 per share in earnings.
Run for president? Start a podcast? Tackle AI? Kamala Harris' options are wide open
Former Vice President Kamala Harris closed a big door when she announced Wednesday that she would not run for California governor. But she left open a heap of others. Departing presidents, vice presidents, first ladies and failed presidential candidates have pursued a wide variety of paths in the past. Empowered with name recognition and influence but with no official role to fill, they possess the freedom to choose their next adventure. Al Gore took up a cause in global warming, while George W. Bush took up painting.
Far-right extremists using games platforms to radicalise teenagers, report warns
Far-right extremists are using livestream gaming platforms to target and radicalise teenage players, a report has warned. The new research, published in the journal Frontiers in Psychology, reveals how a range of extremist groups and individuals use platforms that allow users to chat and livestream while playing video games to recruit and radicalise vulnerable users, mainly young males. UK crime and counter-terror agencies have urged parents to be especially alert to online offenders targeting youngsters during the summer holidays. In an unprecedented move, last week Counter Terrorism Policing, MI5 and the National Crime Agency issued a joint warning to parents and carers that online offenders "will exploit the school holidays to engage in criminal acts with young people when they know less support is readily available". Dr William Allchorn, a senior research fellow at Anglia Ruskin University's international policing and public protection research institute, who carried out the study with his colleague Dr Elisa Orofino, said "gaming-adjacent" platforms were being used as "digital playgrounds" for extremist activity.
A record-breaking lightning bolt just 'shocked' meteorologists
Breakthroughs, discoveries, and DIY tips sent every weekday. In October 2017, a single flash of lightning during a thunderstorm streaked across the Great Plains for 515 miles. The flash traveled from eastern Texas all the way to Kansas City--and now into the record books. The World Meteorological Organization (WMO) certified that this megaflash is now the longest single lightning flash in the United States. The massive lightning bolt is detailed in a study published July 31 in the Bulletin of the American Meteorological Society.
Inside the Summit Where China Pitched Its AI Agenda to the World
Three days after the Trump administration published its much-anticipated AI action plan, the Chinese government put out its own AI policy blueprint. Was the timing a coincidence? China's "Global AI Governance Action Plan" was released on July 26, the first day of the World Artificial Intelligence Conference (WAIC), the largest annual AI event in China. Geoffrey Hinton and Eric Schmidt were among the many Western tech industry figures who attended the festivities in Shanghai. Our WIRED colleague Will Knight was also on the scene.