Recent progress in autonomous code generation has fueled excitement around AI agents capable of accelerating scientific discovery by running experiments. However, there is currently no benchmark that evaluates whether such agents can implement scientific ideas when given varied amounts of code as a starting point, interpolating between reproduction (running code) and from-scratch replication (fully re-implementing and running code). We introduce AutoExperiment, a benchmark that evaluates AI agents' ability to implement and run machine learning experiments based on natural language descriptions in research papers. In each task, agents are given a research paper, a codebase with key functions masked out, and a command to run the experiment. The goal is to generate the missing code, execute the experiment in a sandboxed environment, and reproduce the results. AutoExperiment scales in difficulty by varying the number of missing functions $n$, ranging from partial reproduction to full replication. We evaluate state-of-the-art agents and find that performance degrades rapidly as $n$ increases. Agents that can dynamically interact with the environment (e.g. to debug their code) can outperform agents in fixed "agentless" harnesses, and there exists a significant gap between single-shot and multi-trial success rates (Pass@1 vs. Pass@5), motivating verifier approaches to our benchmark. Our findings highlight critical challenges in long-horizon code generation, context retrieval, and autonomous experiment execution, establishing AutoExperiment as a new benchmark for evaluating progress in AI-driven scientific experimentation. Our data and code are open-sourced at https://github.com/j1mk1m/AutoExperiment .

The article "Revelations: A Decidable Class of POMDP with Omega-Regular Objectives" won an Outstanding Paper Award at the AAAI 2025 conference, a prestigious international conference about artificial intelligence. This year, only three papers received such an award out of 3,000 accepted and 12,000 submitted! This recognition crowns the results of research initiated in Bordeaux (France) within the Synthèse team at the Bordeaux Computer Science Research Laboratory (LaBRI), where four of the authors work: Marius Belly, Nathanaël Fijalkow, Hugo Gimbert, and Pierre Vandenhove. The work also involved researchers from Paris (Florian Horn) and Antwerp (Guillermo A. Pérez). The article is freely available on arXiv, and this post outlines its main ideas.

Chenghua Lu received the B.S. degree in Mechanical Engineering from Northeastern University, Shenyang, China, in 2017, and the M.S. degree in Mechanical Manufacturing and Automation from the University of Chinese Academy of Sciences, Beijing, China, in 2021. She is currently working toward the Ph.D. degree majoring in Engineering Mathematics with the School of Mathematics Engineering and Technology and Bristol Robotics Laboratory, University of Bristol, Bristol, UK. Her research interests include tactile sensing and soft robotics. Kailuan T ang received a B.S. degree in Communication Engineering from the Southern University of Science and Technology (SUSTech), Shenzhen, China in 2017. He is currently working towards a Ph.D. degree majoring in Mechanics with the School of Mechatronics Engineering, Harbin Institute of Technology.

On June 12, Alexandr Wang stepped down as Scale's CEO to chase his most ambitious moonshot yet: building smarter-than-human AI as head of Meta's new "superintelligence" division. As part of his move, Meta will invest 14.3 billion for a minority stake in Scale AI, but the real prize isn't his company--it's Wang himself. Wang, 28, is expected to bring a sense of urgency to Meta's AI efforts, which this year have been plagued by delays and underwhelming performance. Once the undisputed leader of open-weight AI, the U.S. tech giant has been overtaken by Chinese rivals like DeepSeek on popular benchmarks. Although Wang, who dropped out of MIT at 19, lacks the academic chops of some of his peers, he offers both insight into the types of data Meta's rivals use to improve their AI systems, and unrivaled ambition.

This week, Emily Bazelon, John Dickerson, and David Plotz discuss whether the US should join Israel's war on Iran, the tragic Minnesota assassinations and why US political violence is surging now, and the Supreme Court's unsurprising but willfully obtuse decision to uphold Tennessee's youth transgender care ban. Here are some notes and references from this week's show: Alexander Ward, Lara Seligman, and Dustin Volz for The Wall Street Journal (Exclusive): Israel Built Its Case for War With Iran on New Intelligence. The U.S. Didn't Buy It. Thomas L. Friedman for The New York Times (Opinion): The Smart Way for Trump to End the Israel-Iran War Oren Cass for Understanding America (Substack): Is Israel the Ideal "America First" Ally? Warren P. Strobel, Alex Horton, and Abigail Hauslohner for the Washington Post: Navigating Iran crisis, Trump relies on experience over star power Amy Howe for SCOTUSblog: Court upholds Tennessee's ban on certain medical treatments for transgender minors Abbie VanSickle for The New York Times: Sotomayor Writes the Court'Abandons' Transgender Children to'Political Whims' Ella Lee for The Hill: Clarence Thomas urges courts to end deferring to'experts' on gender-affirming care Ian Millhiser for Vox: The Supreme Court's incoherent new attack on trans rights, explained Here are this week's chatters: Emily: A Family Matter by Claire Lynch; The Fall of Affirmative Action: Race, the Supreme Court, and the Future of Higher Education by Justin Driver; A Flower Traveled in My Blood: The Incredible True Story of the Grandmothers Who Fought to Find a Stolen Generation of Children by Haley Cohen Gilliland. John: Mary Cunningham for CBS News: Federal Reserve holds its benchmark interest rate steady at today's FOMC meeting; ABA Banking Journal: Fed's Powell says some areas of U.S. may be'uninsurable' in next decade David: Trip Gabriel for the New York Times: William Langewiesche, the'Steve McQueen of Journalism,' Dies at 70 For this week's Slate Plus bonus episode, Emily, John, and David discuss the exciting possibilities and likely limitations of using AI tools for historical research and writing.

In an attempt to tackle the UN SDGs, AI for Social Impact (AI4SI) projects focus on harnessing AI to address societal issues in areas such as healthcare, social justice, etc. Unfortunately, despite growing interest in AI4SI, achieving tangible, on-the-ground impact remains a significant challenge. For example, identifying and engaging motivated collaborators who are willing to co-design and deploy AI based solutions in real-world settings is often difficult. Even when such partnerships are established, many AI4SI projects "fail" to progress beyond the proof-of-concept stage, and hence, are unable to transition to at-scale production-level solutions. Furthermore, the unique challenges faced by AI4SI researchers are not always fully recognized within the broader AI community, where such work is sometimes viewed as primarily applied and not aligning with the traditional criteria for novelty emphasized in core AI venues. This paper attempts to shine a light on the diverse challenges faced in AI4SI research by diagnosing a multitude of factors that prevent AI4SI partnerships from achieving real-world impact on the ground. Drawing on semi-structured interviews with six leading AI4SI researchers - complemented by the authors' own lived experiences in conducting AI4SI research - this paper attempts to understand the day-to-day difficulties faced in developing and deploying socially impactful AI solutions. Through thematic analysis, we identify structural and organizational, communication, collaboration, and operational challenges as key barriers to deployment. While there are no easy fixes, we synthesize best practices and actionable strategies drawn from these interviews and our own work in this space. In doing so, we hope this paper serves as a practical reference guide for AI4SI researchers and partner organizations seeking to engage more effectively in socially impactful AI collaborations.

Family caregivers of individuals with Alzheimer's Disease and Related Dementia (AD/ADRD) face significant emotional and logistical challenges that place them at heightened risk for stress, anxiety, and depression. Although recent advances in generative AI -- particularly large language models (LLMs) -- offer new opportunities to support mental health, little is known about how caregivers perceive and engage with such technologies. To address this gap, we developed Carey, a GPT-4o-based chatbot designed to provide informational and emotional support to AD/ADRD caregivers. Using Carey as a technology probe, we conducted semi-structured interviews with 16 family caregivers following scenario-driven interactions grounded in common caregiving stressors. Through inductive coding and reflexive thematic analysis, we surface a systemic understanding of caregiver needs and expectations across six themes -- on-demand information access, emotional support, safe space for disclosure, crisis management, personalization, and data privacy. For each of these themes, we also identified the nuanced tensions in the caregivers' desires and concerns. We present a mapping of caregiver needs, AI chatbot's strengths, gaps, and design recommendations. Our findings offer theoretical and practical insights to inform the design of proactive, trustworthy, and caregiver-centered AI systems that better support the evolving mental health needs of AD/ADRD caregivers.

Daniel Rausch, Amazon's vice president of Alexa and Echo, is in the midst of a major transition. More than a decade beyond the launch of Amazon's Alexa, he's been tasked with creating a new version of the marquee voice assistant, one that's powered by large language models. As he put it in my interview with him, this new assistant, dubbed Alexa, is "a complete rebuild of the architecture." How did his team approach Amazon's largest ever revamp of its voice assistant? They used AI to build AI, of course.

The annual RoboCup event, where teams gather from across the globe to take part in competitions across a number of leagues, will this year take place in Brazil, from 15-21 July. An important part of the week is RoboCupJunior, which is designed to introduce RoboCup to school children, and sees hundreds of kids taking part in a variety of challenges across different leagues. This year, the lead organizer for RoboCupJunior is Ana Patrícia Magalhães. We caught up with her to find out how the preparations are going, what to expect at this year's competition, and how RoboCup inspires communities. RoboCup will take place from 15-21 July, in Salvador, Brazil.

Despite promising developments in Explainable Artificial Intelligence, the practical value of XAI methods remains under-explored and insufficiently validated in real-world settings. Robust and context-aware evaluation is essential, not only to produce understandable explanations but also to ensure their trustworthiness and usability for intended users, but tends to be overlooked because of no clear guidelines on how to design an evaluation with users. This study addresses this gap with two main goals: (1) to develop a framework of well-defined, atomic properties that characterise the user experience of XAI in healthcare; and (2) to provide clear, context-sensitive guidelines for defining evaluation strategies based on system characteristics. We conducted a systematic review of 82 user studies, sourced from five databases, all situated within healthcare settings and focused on evaluating AI-generated explanations. The analysis was guided by a predefined coding scheme informed by an existing evaluation framework, complemented by inductive codes developed iteratively. The review yields three key contributions: (1) a synthesis of current evaluation practices, highlighting a growing focus on human-centred approaches in healthcare XAI; (2) insights into the interrelations among explanation properties; and (3) an updated framework and a set of actionable guidelines to support interdisciplinary teams in designing and implementing effective evaluation strategies for XAI systems tailored to specific application contexts.