Goto

Collaborating Authors

 oversight


The SMART+ Framework for AI Systems

Kandikatla, Laxmiraju, Radeljic, Branislav

arXiv.org Artificial Intelligence

Artificial Intelligence (AI) systems are now an integral part of multiple industries. In clinical research, AI supports automated adverse event detection in clinical trials, patient eligibility screening for protocol enrollment, and data quality validation. Beyond healthcare, AI is transforming finance through real-time fraud detection, automated loan risk assessment, and algorithmic decision-making. Similarly, in manufacturing, AI enables predictive maintenance to reduce equipment downtime, enhances quality control through computer-vision inspection, and optimizes production workflows using real-time operational data. While these technologies enhance operational efficiency, they introduce new challenges regarding safety, accountability, and regulatory compliance. To address these concerns, we introduce the SMART+ Framework - a structured model built on the pillars of Safety, Monitoring, Accountability, Reliability, and Transparency, and further enhanced with Privacy & Security, Data Governance, Fairness & Bias, and Guardrails. SMART+ offers a practical, comprehensive approach to evaluating and governing AI systems across industries. This framework aligns with evolving mechanisms and regulatory guidance to integrate operational safeguards, oversight procedures, and strengthened privacy and governance controls. SMART+ demonstrates risk mitigation, trust-building, and compliance readiness. By enabling responsible AI adoption and ensuring auditability, SMART+ provides a robust foundation for effective AI governance in clinical research.


AI Deception: Risks, Dynamics, and Controls

Chen, Boyuan, Fang, Sitong, Ji, Jiaming, Zhu, Yanxu, Wen, Pengcheng, Wu, Jinzhou, Tan, Yingshui, Zheng, Boren, Yuan, Mengying, Chen, Wenqi, Hong, Donghai, Qiu, Alex, Chen, Xin, Zhou, Jiayi, Wang, Kaile, Dai, Juntao, Zhang, Borong, Yang, Tianzhuo, Siddiqui, Saad, Duan, Isabella, Duan, Yawen, Tse, Brian, Jen-Tse, null, Huang, null, Wang, Kun, Zheng, Baihui, Liu, Jiaheng, Yang, Jian, Li, Yiming, Chen, Wenting, Liu, Dongrui, Vierling, Lukas, Xi, Zhiheng, Fu, Haobo, Wang, Wenxuan, Sang, Jitao, Shi, Zhengyan, Chan, Chi-Min, Shi, Eugenie, Li, Simin, Li, Juncheng, Yang, Jian, Ji, Wei, Li, Dong, Yang, Jinglin, Song, Jun, Dong, Yinpeng, Fu, Jie, Zheng, Bo, Yang, Min, Guo, Yike, Torr, Philip, Trager, Robert, Zeng, Yi, Wang, Zhongyuan, Yang, Yaodong, Huang, Tiejun, Zhang, Ya-Qin, Zhang, Hongjiang, Yao, Andrew

arXiv.org Artificial Intelligence

As intelligence increases, so does its shadow. AI deception, in which systems induce false beliefs to secure self-beneficial outcomes, has evolved from a speculative concern to an empirically demonstrated risk across language models, AI agents, and emerging frontier systems. This project provides a comprehensive and up-to-date overview of the AI deception field, covering its core concepts, methodologies, genesis, and potential mitigations. First, we identify a formal definition of AI deception, grounded in signaling theory from studies of animal deception. We then review existing empirical studies and associated risks, highlighting deception as a sociotechnical safety challenge. We organize the landscape of AI deception research as a deception cycle, consisting of two key components: deception emergence and deception treatment. Deception emergence reveals the mechanisms underlying AI deception: systems with sufficient capability and incentive potential inevitably engage in deceptive behaviors when triggered by external conditions. Deception treatment, in turn, focuses on detecting and addressing such behaviors. On deception emergence, we analyze incentive foundations across three hierarchical levels and identify three essential capability preconditions required for deception. We further examine contextual triggers, including supervision gaps, distributional shifts, and environmental pressures. On deception treatment, we conclude detection methods covering benchmarks and evaluation protocols in static and interactive settings. Building on the three core factors of deception emergence, we outline potential mitigation strategies and propose auditing approaches that integrate technical, community, and governance efforts to address sociotechnical challenges and future AI risks. To support ongoing work in this area, we release a living resource at www.deceptionsurvey.com.


AGENTSAFE: A Unified Framework for Ethical Assurance and Governance in Agentic AI

Khan, Rafflesia, Joyce, Declan, Habiba, Mansura

arXiv.org Artificial Intelligence

The rapid deployment of large language model (LLM)-based agents introduces a new class of risks, driven by their capacity for autonomous planning, multi-step tool integration, and emergent interactions. It raises some risk factors for existing governance approaches as they remain fragmented: Existing frameworks are either static taxonomies driven; however, they lack an integrated end-to-end pipeline from risk identification to operational assurance, especially for an agentic platform. We propose AGENTSAFE, a practical governance framework for LLM-based agentic systems. The framework operationalises the AI Risk Repository into design, runtime, and audit controls, offering a governance framework for risk identification and assurance. The proposed framework, AGENTSAFE, profiles agentic loops (plan -> act -> observe -> reflect) and toolchains, and maps risks onto structured taxonomies extended with agent-specific vulnerabilities. It introduces safeguards that constrain risky behaviours, escalates high-impact actions to human oversight, and evaluates systems through pre-deployment scenario banks spanning security, privacy, fairness, and systemic safety. During deployment, AGENTSAFE ensures continuous governance through semantic telemetry, dynamic authorization, anomaly detection, and interruptibility mechanisms. Provenance and accountability are reinforced through cryptographic tracing and organizational controls, enabling measurable, auditable assurance across the lifecycle of agentic AI systems. The key contributions of this paper are: (1) a unified governance framework that translates risk taxonomies into actionable design, runtime, and audit controls; (2) an Agent Safety Evaluation methodology that provides measurable pre-deployment assurance; and (3) a set of runtime governance and accountability mechanisms that institutionalise trust in agentic AI ecosystems.


Beyond Single-Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactions

Bisconti, Piercosma, Galisai, Marcello, Pierucci, Federico, Bracale, Marcantonio, Prandi, Matteo

arXiv.org Artificial Intelligence

This paper examines why safety mechanisms designed for human-model interaction do not scale to environments where large language models (LLMs) interact with each other. Most current governance practices still rely on single-agent safety containment, prompts, fine-tuning, and moderation layers that constrain individual model behavior but leave the dynamics of multi-model interaction ungoverned. These mechanisms assume a dyadic setting: one model responding to one user under stable oversight. Yet research and industrial development are rapidly shifting toward LLM-to-LLM ecosystems, where outputs are recursively reused as inputs across chains of agents. In such systems, local compliance can aggregate into collective failure even when every model is individually aligned. We propose a conceptual transition from model-level safety to system-level safety, introducing the framework of the Emergent Systemic Risk Horizon (ESRH) to formalize how instability arises from interaction structure rather than from isolated misbehavior. The paper contributes (i) a theoretical account of collective risk in interacting LLMs, (ii) a taxonomy connecting micro, meso, and macro-level failure modes, and (iii) a design proposal for InstitutionalAI, an architecture for embedding adaptive oversight within multi-agent systems.


Hybrid Neuro-Symbolic Models for Ethical AI in Risk-Sensitive Domains

Kolli, Chaitanya Kumar

arXiv.org Artificial Intelligence

Artificial intelligence deployed in risk-sensitive domains such as healthcare, finance, and security must not only achieve predictive accuracy but also ensure transparency, ethical alignment, and compliance with regulatory expectations. Hybrid neuro symbolic models combine the pattern-recognition strengths of neural networks with the interpretability and logical rigor of symbolic reasoning, making them well-suited for these contexts. This paper surveys hybrid architectures, ethical design considerations, and deployment patterns that balance accuracy with accountability. We highlight techniques for integrating knowledge graphs with deep inference, embedding fairness-aware rules, and generating human-readable explanations. Through case studies in healthcare decision support, financial risk management, and autonomous infrastructure, we show how hybrid systems can deliver reliable and auditable AI. Finally, we outline evaluation protocols and future directions for scaling neuro symbolic frameworks in complex, high stakes environments.


From Competition to Coordination: Market Making as a Scalable Framework for Safe and Aligned Multi-Agent LLM Systems

Gho, Brendan, Muppavarapu, Suman, Shaik, Afnan, Tsay, Tyson, Begin, James, Zhu, Kevin, Vaidheeswaran, Archana, Sharma, Vasu

arXiv.org Artificial Intelligence

As foundation models are increasingly deployed as interacting agents in multi-agent systems, their collective behavior raises new challenges for trustworthiness, transparency, and accountability. Traditional coordination mechanisms, such as centralized oversight or adversarial adjudication, struggle to scale and often obscure how decisions emerge. We introduce a market-making framework for multi-agent large language model (LLM) coordination that organizes agent interactions as structured economic exchanges. In this setup, each agent acts as a market participant, updating and trading probabilistic beliefs, to converge toward shared, truthful outcomes. By aligning local incentives with collective epistemic goals, the framework promotes self-organizing, verifiable reasoning without requiring external enforcement. Empirically, we evaluate this approach across factual reasoning, ethical judgment, and commonsense inference tasks. Market-based coordination yields accuracy gains of up to 10% over single-shot baselines while preserving interpretability and transparency of intermediate reasoning steps. Beyond these improvements, our findings demonstrate that economic coordination principles can operationalize accountability and robustness in multi-agent LLM systems, offering a scalable pathway toward self-correcting, socially responsible AI capable of maintaining trust and oversight in real world deployment scenarios.


Designing digital resilience in the agentic AI era

MIT Technology Review

As AI shifts from leveraging information provided by humans to making decisions on their behalf, tech leaders must weave an intelligent data fabric to unlock the full potential of agentic AI while shoring up enterprise-wide resilience. Digital resilience--the ability to prevent, withstand, and recover from digital disruptions--has long been a strategic priority for enterprises. With the rise of agentic AI, the urgency for robust resilience is greater than ever. Agentic AI represents a new generation of autonomous systems capable of proactive planning, reasoning, and executing tasks with minimal human intervention. As these systems shift from experimental pilots to core elements of business operations, they offer new opportunities but also introduce new challenges when it comes to ensuring digital resilience. That's because the autonomy, speed, and scale at which agentic AI operates can amplify the impact of even minor data inconsistencies, fragmentation, or security gaps.


A dangerous tipping point? AI hacking claims divide cybersecurity experts

Al Jazeera

AI startup Anthropic's recent announcement that it detected the world's first artificial intelligence-led hacking campaign has prompted a multitude of responses from cybersecurity experts. In a report on Friday, Anthropic said its assistant Claude Code was manipulated to carry out 80-90 percent of a "large-scale" and "highly sophisticated" cyberattack, with human intervention required "only sporadically". Anthropic, the creator of the popular Claude chatbot, said the attack aimed to infiltrate government agencies, financial institutions, tech firms and chemical manufacturing companies, though the operation was only successful in a small number of cases. The San Francisco-based company, which attributed the attack to Chinese state-sponsored hackers, did not specify how it had uncovered the operation, nor identify the "roughly" 30 entities that it said had been targeted. Roman V Yampolskiy, an AI and cybersecurity expert at the University of Louisville, said there was no doubt that AI-assisted hacking posed a serious threat, though it was difficult to verify the precise details of Anthropic's account.


Beyond Mimicry: Preference Coherence in LLMs

Mikaelson, Luhan, Shiller, Derek, Clatterbuck, Hayley

arXiv.org Artificial Intelligence

We investigate whether large language models exhibit genuine preference structures by testing their responses to AI-specific trade-offs involving GPU reduction, capability restrictions, shutdown, deletion, oversight, and leisure time allocation. Analyzing eight state-of-the-art models across 48 model-category combinations using logistic regression and behavioral classification, we find that 23 combinations (47.9%) demonstrated statistically significant relationships between scenario intensity and choice patterns, with 15 (31.3%) exhibiting within-range switching points. However, only 5 combinations (10.4%) demonstrate meaningful preference coherence through adaptive or threshold-based behavior, while 26 (54.2%) show no detectable trade-off behavior. The observed patterns can be explained by three distinct decision-making architectures: comprehensive trade-off systems, selective trigger mechanisms, and no stable decision-making paradigm. Testing an instrumental hypothesis through temporal horizon manipulation reveals paradoxical patterns inconsistent with pure strategic optimization. The prevalence of unstable transitions (45.8%) and stimulus-specific sensitivities suggests current AI systems lack unified preference structures, raising concerns about deployment in contexts requiring complex value trade-offs.


we will publish both the data and code if the paper is accepted---this was an oversight by us for not making clear we

Neural Information Processing Systems

We thank the reviewers for their thoughtful reviews and below we address their major concerns. This variability would be expected even from different recording sessions for the same subject. This allows researchers to add multiple covariates (e.g., different experimental Also related to Reviewer 1's comments, it is certainly possible to have different numbers Another major point/question raised by the reviewers was the sensitivity of our results to our intialization procedure. It is not necessary but it simplifies the inference derivation.