Goto

Collaborating Authors

 Large Language Model


On the Effectiveness of Large Language Models in Writing Alloy Formulas

arXiv.org Artificial Intelligence

Declarative specifications have a vital role to play in developing safe and dependable software systems. Writing specifications correctly, however, remains particularly challenging. This paper presents a controlled experiment on using large language models (LLMs) to write declarative formulas in the well-known language Alloy. Our use of LLMs is three-fold. One, we employ LLMs to write complete Alloy formulas from given natural language descriptions (in English). Two, we employ LLMs to create alternative but equivalent formulas in Alloy with respect to given Alloy formulas. Three, we employ LLMs to complete sketches of Alloy formulas and populate the holes in the sketches by synthesizing Alloy expressions and operators so that the completed formulas accurately represent the desired properties (that are given in natural language). We conduct the experimental evaluation using 11 well-studied subject specifications and employ two popular LLMs, namely ChatGPT and DeepSeek. The experimental results show that the LLMs generally perform well in synthesizing complete Alloy formulas from input properties given in natural language or in Alloy, and are able to enumerate multiple unique solutions. Moreover, the LLMs are also successful at completing given sketches of Alloy formulas with respect to natural language descriptions of desired properties (without requiring test cases). We believe LLMs offer a very exciting advance in our ability to write specifications, and can help make specifications take a pivotal role in software development and enhance our ability to build robust software.


Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs

arXiv.org Artificial Intelligence

Software testing remains the most widely used methodology for validating quality of code. However, effectiveness of testing critically depends on the quality of test suites used. Test cases in a test suite consist of two fundamental parts: (1) input values for the code under test, and (2) correct checks for the outputs it produces. These checks are commonly written as assertions, and termed test oracles. The last couple of decades have seen much progress in automated test input generation, e.g., using fuzzing and symbolic execution. However, automating test oracles remains a relatively less explored problem area. Indeed, a test oracle by its nature requires knowledge of expected behavior, which may only be known to the developer and may not not exist in a formal language that supports automated reasoning. Our focus in this paper is automation of test oracles for clients of widely used Java libraries, e.g., java.lang and java.util packages. Our key insight is that Javadocs that provide a rich source of information can enable automated generation of test oracles. Javadocs of the core Java libraries are fairly detailed documents that contain natural language descriptions of not only how the libraries behave but also how the clients must (not) use them. We use large language models as an enabling technology to embody our insight into a framework for test oracle automation, and evaluate it experimentally. Our experiments demonstrate that LLMs can generate oracles for checking normal and exceptional behaviors from Javadocs, with 98.8% of these oracles being compilable and 96.4% accurately reflecting intended properties. Even for the few incorrect oracles, errors are minor and can be easily corrected with the help of additional comment information generated by the LLMs.


OpenAI signs deal with UK to find government uses for its models

The Guardian

Sam Altman, leader of one of the world's biggest artificial intelligence companies, has signed a deal with the British government to explore the deployment of advanced AI models in areas including justice, security and education. The chief executive of OpenAI, which has been valued at 300bn ( 220bn) and provides the ChatGPT suite of large language models, agreed the memorandum of understanding with the science and technology secretary, Peter Kyle, on Monday. It follows a similarly wide-ranging deal between the UK government and OpenAI's rival US tech company, Google, which campaigners called "dangerously naive", citing fears that the arrangement could leave the public sector dependent on private technology providers and make it harder for politicians to regulate them. The latest agreement states that OpenAI and the government "will collaborate to identify opportunities for how advanced AI models can be deployed throughout government", including "to help civil servants work more efficiently" and to support "citizens to navigate public services more effectively". It said they will collaborate to develop AI solutions "to the UK's hardest problems, including in areas such as justice, defence and security, and education technology" and develop partnerships "to expand public engagement with AI technology".


Meta builds world's largest AI superclusters for the future

FOX News

The CyberGuy Kurt Knutsson joins'Fox & Friends' to discuss the U.S.-Saudi investment summit and the debate over regulation as artificial intelligence continues to advance. What happens when one of the world's richest companies decides to go all-in on artificial intelligence? If you're Meta Platforms CEO Mark Zuckerberg, it means launching superclusters so large they could rival the footprint of Manhattan. Recently, Zuckerberg unveiled plans to invest "hundreds of billions of dollars" into next-generation AI infrastructure, including some of the largest compute clusters the world has ever seen. Meta's first supercluster, called Prometheus, is slated to go live in 2026.


OpenAI's New CEO of Applications Strikes Hyper-Optimistic Tone in First Memo to Staff

WIRED

OpenAI's incoming CEO of applications, Fidji Simo, sent her first note to staff on Monday, telling employees the tools they're developing "will unlock more opportunities for more people than any other technology in history." "If we get this right, AI can give everyone more power than ever," Simo wrote, striking a hyper-optimistic tone, according to a copy of the memo viewed by WIRED. "But I also realize those opportunities won't magically appear on their own." Simo previously worked as the CEO of Instacart. Before that, she spent a decade at Meta, where she went from being a product manager on the company's news feed to the head of product for the Facebook app.


Add several features to Chrome with this free DeepSeek AI extension

PCWorld

Last year, DeepSeek turned all kinds of heads when it came on the scene and presented itself as a faster, more-efficient alternative to options like ChatGPT and Copilot, all available for free. Some of that hype has worn off since, but DeepSeek is still up there as a top AI chatbot. If you're using Google Chrome or any other Chromium-based browser--including Microsoft Edge, Opera, and Vivaldi--then you can actually use DeepSeek directly in your browser via the DeepSeek Assistant extension, available in the Chrome Web Store. Here you'll find the input window for making DeepSeek requests. Type in your prompt and submit via the blue paper plane icon, and you'll get a response shortly after.


Human-level AI is not inevitable. We have the power to change course Garrison Lovely

The Guardian

"Technology happens because it is possible," OpenAI CEO, Sam Altman, told the New York Times in 2019, consciously paraphrasing Robert Oppenheimer, the father of the atomic bomb. Another widespread techie conviction is that the first human-level AI โ€“ also known as artificial general intelligence (AGI) โ€“ will lead to one of two futures: a post-scarcity techno-utopia or the annihilation of humanity. For countless other species, the arrival of humans spelled doom. We weren't tougher, faster or stronger โ€“ just smarter and better coordinated. In many cases, extinction was an accidental byproduct of some other goal we had.


LLaPipe: LLM-Guided Reinforcement Learning for Automated Data Preparation Pipeline Construction

arXiv.org Artificial Intelligence

Automated data preparation is crucial for democratizing machine learning, yet existing reinforcement learning (RL) based approaches suffer from inefficient exploration in the vast space of possible preprocessing pipelines. We present LLaPipe, a novel framework that addresses this exploration bottleneck by integrating Large Language Models (LLMs) as intelligent policy advisors. Unlike traditional methods that rely solely on statistical features and blind trial-and-error, LLaPipe leverages the semantic understanding capabilities of LLMs to provide contextually relevant exploration guidance. Our framework introduces three key innovations: (1) an LLM Policy Advisor that analyzes dataset semantics and pipeline history to suggest promising preprocessing operations, (2) an Experience Distillation mechanism that mines successful patterns from past pipelines and transfers this knowledge to guide future exploration, and (3) an Adaptive Advisor Triggering strategy (Advisor\textsuperscript{+}) that dynamically determines when LLM intervention is most beneficial, balancing exploration effectiveness with computational cost. Through extensive experiments on 18 diverse datasets spanning multiple domains, we demonstrate that LLaPipe achieves up to 22.4\% improvement in pipeline quality and 2.3$\times$ faster convergence compared to state-of-the-art RL-based methods, while maintaining computational efficiency through selective LLM usage (averaging only 19.0\% of total exploration steps).


DPMT: Dual Process Multi-scale Theory of Mind Framework for Real-time Human-AI Collaboration

arXiv.org Artificial Intelligence

Real-time human-artificial intelligence (AI) collaboration is crucial yet challenging, especially when AI agents must adapt to diverse and unseen human behaviors in dynamic scenarios. Existing large language model (LLM) agents often fail to accurately model the complex human mental characteristics such as domain intentions, especially in the absence of direct communication. To address this limitation, we propose a novel dual process multi-scale theory of mind (DPMT) framework, drawing inspiration from cognitive science dual process theory. Our DPMT framework incorporates a multi-scale theory of mind (ToM) module to facilitate robust human partner modeling through mental characteristic reasoning. Experimental results demonstrate that DPMT significantly enhances human-AI collaboration, and ablation studies further validate the contributions of our multi-scale ToM in the slow system.


Automated Interpretation of Non-Destructive Evaluation Contour Maps Using Large Language Models for Bridge Condition Assessment

arXiv.org Artificial Intelligence

Bridge maintenance and safety are essential for transportation authorities, and Non-Destructive Evaluation (NDE) techniques are critical to assessing structural integrity. However, interpreting NDE data can be time-consuming and requires expertise, potentially delaying decision-making. Recent advancements in Large Language Models (LLMs) offer new ways to automate and improve this analysis. This pilot study introduces a holistic assessment of LLM capabilities for interpreting NDE contour maps and demonstrates the effectiveness of LLMs in providing detailed bridge condition analyses. It establishes a framework for integrating LLMs into bridge inspection workflows, indicating that LLM-assisted analysis can enhance efficiency without compromising accuracy. In this study, several LLMs are explored with prompts specifically designed to enhance the quality of image descriptions, which are applied to interpret five different NDE contour maps obtained through technologies for assessing bridge conditions. Each LLM model is evaluated based on its ability to produce detailed descriptions, identify defects, provide actionable recommendations, and demonstrate overall accuracy. The research indicates that four of the nine models provide better image descriptions, effectively covering a wide range of topics related to the bridge's condition. The outputs from these four models are summarized using five different LLMs to form a comprehensive overview of the bridge. Notably, LLMs ChatGPT-4 and Claude 3.5 Sonnet generate more effective summaries. The findings suggest that LLMs have the potential to significantly improve efficiency and accuracy. This pilot study presents an innovative approach that leverages LLMs for image captioning in parallel and summarization, enabling faster decision-making in bridge maintenance and enhancing infrastructure management and safety assessments.