Goto

Collaborating Authors

 Government


MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

arXiv.org Artificial Intelligence

We introduce MCP-Bench, a benchmark for evaluating large language models (LLMs) on realistic, multi-step tasks that demand tool use, cross-tool coordination, precise parameter control, and planning/reasoning for solving tasks. Built on the Model Context Protocol (MCP), MCP-Bench connects LLMs to 28 representative live MCP servers spanning 250 tools across domains such as finance, traveling, scientific computing, and academic search. Unlike prior API-based benchmarks, each MCP server provides a set of complementary tools designed to work together, enabling the construction of authentic, multi-step tasks with rich input-output coupling. Tasks in MCP-Bench test agents' ability to retrieve relevant tools from fuzzy instructions without explicit tool names, plan multi-hop execution trajectories for complex objectives, ground responses in intermediate tool outputs, and orchestrate cross-domain workflows - capabilities not adequately evaluated by existing benchmarks that rely on explicit tool specifications, shallow few-step workflows, and isolated domain operations. We propose a multi-faceted evaluation framework covering tool-level schema understanding and usage, trajectory-level planning, and task completion. Experiments on 20 advanced LLMs reveal persistent challenges in MCP-Bench. Code and data: https://github.com/Accenture/mcp-bench.


Governable AI: Provable Safety Under Extreme Threat Models

arXiv.org Artificial Intelligence

As AI rapidly advances, the security risks posed by AI are becoming increasingly severe, especially in critical scenarios, including those posing existential risks. If AI becomes uncontrollable, manipulated, or actively evades safety mechanisms, it could trigger systemic disasters. Existing AI safety approaches-such as model enhancement, value alignment, and human intervention-suffer from fundamental, in-principle limitations when facing AI with extreme motivations and unlimited intelligence, and cannot guarantee security. To address this challenge, we propose a Governable AI (GAI) framework that shifts from traditional internal constraints to externally enforced structural compliance based on cryptographic mechanisms that are computationally infeasible to break, even for future AI, under the defined threat model and well-established cryptographic assumptions.The GAI framework is composed of a simple yet reliable, fully deterministic, powerful, flexible, and general-purpose rule enforcement module (REM); governance rules; and a governable secure super-platform (GSSP) that offers end-to-end protection against compromise or subversion by AI. The decoupling of the governance rules and the technical platform further enables a feasible and generalizable technical pathway for the safety governance of AI. REM enforces the bottom line defined by governance rules, while GSSP ensures non-bypassability, tamper-resistance, and unforgeability to eliminate all identified attack vectors. This paper also presents a rigorous formal proof of the security properties of this mechanism and demonstrates its effectiveness through a prototype implementation evaluated in representative high-stakes scenarios.


Surveying the Operational Cybersecurity and Supply Chain Threat Landscape when Developing and Deploying AI Systems

arXiv.org Artificial Intelligence

The rise of AI has transformed the software and hardware landscape, enabling powerful capabilities through specialized infrastructures, large-scale data storage, and advanced hardware. However, these innovations introduce unique attack surfaces and objectives which traditional cybersecurity assessments often overlook. Cyber attackers are shifting their objectives from conventional goals like privilege escalation and network pivoting to manipulating AI outputs to achieve desired system effects, such as slowing system performance, flooding outputs with false positives, or degrading model accuracy. This paper serves to raise awareness of the novel cyber threats that are introduced when incorporating AI into a software system. W e explore the operational cybersecurity and supply chain risks across the AI lifecycle, emphasizing the need for tailored security frameworks to address evolving threats in the AI-driven landscape. W e highlight previous exploitations and provide insights from working in this area. By understanding these risks, organizations can better protect AI systems and ensure their reliability and resilience.


Beyond Optimization: Exploring Novelty Discovery in Autonomous Experiments

arXiv.org Artificial Intelligence

Autonomous experiments (AEs) are transforming how scientific research is conducted by integrating artificial intelligence with automated experimental platforms. Current AEs primarily focus on the optimization of a predefined target; while accelerating this goal, such an approach limits the discovery of unexpected or unknown physical phenomena. Here, we introduce a novel framework, INS2ANE (Integrated Novelty Score-Strategic Autonomous Non-Smooth Exploration), to enhance the discovery of novel phenomena in autonomous experimentation. Our method integrates two key components: (1) a novelty scoring system that evaluates the uniqueness of experimental results, and (2) a strategic sampling mechanism that promotes exploration of under-sampled regions even if they appear less promising by conventional criteria. We validate this approach on a pre-acquired dataset with a known ground truth comprising of image-spectral pairs. We further implement the process on autonomous scanning probe microscopy experiments. INS2ANE significantly increases the diversity of explored phenomena in comparison to conventional optimization routines, enhancing the likelihood of discovering previously unobserved phenomena. These results demonstrate the potential for AE to enhance the depth of scientific discovery; in combination with the efficiency provided by AEs, this approach promises to accelerate scientific research by simultaneously navigating complex experimental spaces to uncover new phenomena.


Intel receives 5.7bn as Trump administration buys 10 percent stake

Al Jazeera

The chief financial officer for the chip manufacturer Intel, David Zinsner, has announced his company received 5.7bn as part of a deal negotiated with the administration of United States President Donald Trump. During an investor conference on Thursday, Zinsner said that Intel, a leader in the US development of semiconductor chips, received the funds on Wednesday evening. Last week, the White House revealed the federal government would take a 10 percent stake in the struggling tech giant, based in Santa Clara, California. As part of the deal, the government negotiated a five-year warrant for an additional 5 percent of Intel's shares, in case the company should cease to own more than 51 percent of its manufacturing operations. "I don't think there's a high likelihood that we would take our stake below 50 percent," Zinsner said.


Two Lebanese soldiers killed in Israeli drone explosion in southern Lebanon

Al Jazeera

The Lebanese military says two soldiers have been killed and two wounded as they investigated an Israeli drone crash in southern Lebanon. The army said the downed Israeli drone exploded on Thursday during an inspection at the crash site in the Naqoura area, not far from Lebanon's border with Israel. Lebanese President Joseph Aoun offered condolences to the soldiers who were killed and injured, stressing that the military "is paying, in blood, the price of preserving stability in the south" of the country. The deadly incident came as Israel has been carrying out near-daily attacks on Lebanon despite a ceasefire reached with Hezbollah in November. It also coincides with a United Nations Security Council vote to wind down a UN peacekeeping mission in southern Lebanon, which has for decades been tasked with maintaining a buffer between Hezbollah fighters and Israeli forces.


Inside Mars, a 'rocky road' mantle reveals a violent past

Popular Science

Breakthroughs, discoveries, and DIY tips sent every weekday. Don't let the appetizing description fool you. When planetary scientists say the interior of Mars resembles a rocky road brownie more than a piece of buttery shortbread, the tasty metaphor masks billions of years of geological violence. In a re-examination of previous observations collected by NASA's decommissioned InSight probe, researchers have discovered that the Martian mantle is embedded with ancient fragments measuring as much as 2.5 miles wide. The data is detailed in a study published on August 28 in Nature.


Rescuers dig through rubble for victims of Russian attack in Kyiv

Al Jazeera

An overnight Russian drone and missile attack on Ukraine's capital has killed at least 14 people including three children, say officials. Al Jazeera's Zein Basravi has been at the scene of a strike on an apartment building there.


Trump admin plans to impose 4-year limits for foreign students studying in US

FOX News

The Trump administration on Wednesday announced a proposed rule to limit the length of time international students can remain in the U.S. for their studies to four years. If finalized, the proposed rule set to be published on Thursday would limit how long certain visa holders, including foreign students, are allowed to stay in the U.S., according to a press release from the Department of Homeland Security, which said the proposal seeks to curb "visa abuse" and increase the agency's ability to "properly vet and oversee these individuals." The agency said foreign students have "taken advantage of U.S. generosity" and become "forever students" by remaining enrolled in colleges so they could stay in the U.S. The Trump administration announced a proposed rule to limit the length of time international students can remain in the U.S. for their studies to four years. "For too long, past Administrations have allowed foreign students and other visa holders to remain in the U.S. virtually indefinitely, posing safety risks, costing untold amount of taxpayer dollars, and disadvantaging U.S. citizens," a DHS spokesperson said in a statement. "This new proposed rule would end that abuse once and for all by limiting the amount of time certain visa holders are allowed to remain in the U.S., easing the burden on the federal government to properly oversee foreign students and their history," the spokesperson continued. Since 1978, foreign students, or F visa holders, have been allowed to remain in the U.S. for their "duration of status," which means the time they were enrolled as a full-time student.


Child among three killed in Russian drone and missile attack on Kyiv

BBC News

The latest international effort to achieve a ceasefire in Ukraine was launched by US President Donald Trump earlier this month. He met his Russian counterpart Vladimir Putin in Alaska and Zelensky with European leaders in Washington. Trump has been pushing for a Putin-Zelensky summit. Ukraine's president has backed the move, but he has sought security guarantees from Western allies to prevent any future Russian attack in the event of a peace deal. On Tuesday, Zelensky met the head of Britain's armed forces, Adm Sir Tony Radakin, in Kyiv, where they discussed efforts to end the war.