In this paper we explore hallucinations and related capability limitations in LLMs and LLM - based agents from the perspective of computational complexity . We show that beyond a certain complexity, LLMs are incapable of carrying out computational and agentic tasks or verifying the ir accuracy . Introduction With widespread adoption of transformer - based language models ("LLMs") in AI, there is significant interest in the limits of LLMs' capabilities, specifically so - called "hallucinations", occurrences in which LLMs provide spurious, factually incorrect or nonsensical [1, 2] information when prompted on certain subjects. Further more, there is growing interest in "agentic" uses of LLMs - that is, using LLMs to create "agents" that act autonomously or semi - autonomously to carry out various tasks, including tasks with applications in the real world. This makes it important to understand the types of tasks LLMs can and cannot perform.

Thinking Machines Lab, an artificial intelligence company founded by top researchers who fled OpenAI, has raised a record 2 billion seed round that values the fledgling firm at 12 billion. The funding round was led by Andreessen Horowitz and included Nvidia, Accel, Cisco, and AMD--among others. The mammoth investment reflects the ultracompetitive race to build advanced AI systems, as well as the premium placed on top AI talent. It is the largest seed funding round in history. Thinking Machines is led by CEO Mira Murati, who stepped down as OpenAI's chief technology officer last September.

Sign up for the Slatest to get the most insightful analysis, criticism, and advice out there, delivered to your inbox daily. As a classically trained singer preparing for a professional career, Erin Perry can see quite clearly how artificial intelligence is upending her field--all the way down to the classroom. Perry just completed her first year as a graduate student in voice performance at the Peabody Institute, the prestigious music conservatory run by Johns Hopkins University. It's been rewarding so far: She's been learning how to navigate the modern classical music sector and confronting the relevant impacts of generative A.I., having taken on a project to study the major record labels' lawsuit against the Amazon-backed A.I. startup Anthropic, which trained its models on songwriters' lyrics sans permission or compensation. Understandably, Perry's rather skeptical of A.I.'s artistic applications, and fearful of the sweeping effects it could have on her chosen field, especially as generative-music startups like Suno and Udio are programmed to replicate specific artists and musical styles.

AI companies generally keep a tight grip on their models to discourage misuse. For example, if you ask ChatGPT to give you someone's phone number or instructions for doing something illegal, it will likely just tell you it cannot help. However, as many examples over time have shown, clever prompt engineering or model fine-tuning can sometimes get these models to say things they otherwise wouldn't. The unwanted information may still be hiding somewhere inside the model so that it can be accessed with the right techniques. At present, companies tend to deal with this issue by applying guardrails; the idea is to check whether the prompts or the AI's responses contain disallowed material.

The companies could face an uphill battle. Right now, most of the public perceives AI's use in the classroom as nothing short of ruinous--a surefire way to dampen critical thinking and hasten the decline of our collective attention span (a viral story from New York magazine, for example, described how easy it now is to coast through college thanks to constant access to ChatGPT). Amid that onslaught, AI companies insist that AI promises more individualized learning, faster and more creative lesson planning, and quicker grading. The companies sponsoring this initiative are, of course, not doing it out of the goodness of their hearts. No--as they hunt for profits, their goal is to make users out of teachers and students.

Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages, largely due to limited exposure to these languages during pre-training. A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data. Among such methods, OFA (Liu et al., 2024a) proposes a similarity-based subword embedding initialization heuristic that is both effective and efficient. However, OFA restricts target-language token embeddings to be convex combinations of a fixed number of source-language embeddings, which may limit expressiveness. To overcome this limitation, we propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding initialization. The hypernetwork is trained to map from an external multilingual word vector space to the PLMs token embedding space using source-language tokens. Once trained, it can generate flexible embeddings for target-language tokens, serving as a good starting point for continual pretraining. Experiments demonstrate that HYPEROFA consistently outperforms random initialization baseline and matches or exceeds the performance of OFA in both continual pre-training convergence and downstream task performance. We make the code publicly available.

In 2021, OpenAI shocked the world by improving the zero-shot classification accuracy on ImageNet from 11.5% to 76.2% via the CLIP series of models (Radford et al., 2021). This event redefined the goal of zero-shot prediction from producing models that generalized to unseen classes to those that generalized to unseen tasks entirely. Two fundamental drivers of CLIP's success were 1) the use of natural language as a medium for representing arbitrary classes (as in the previous state-of-the-art Visual N-grams (Li et al., 2017)), and 2) a massive, yet carefully designed pre-training set which significantly impacted downstream performance Radford et al. (2021); Fang et al. (2023); Xu et al. (2024). Despite the remarkable success of these foundation model-based pipelines Bommasani et al. (2022), there are unique components of zero-shot prediction that warrant investigation from a theoretical point of view. To clarify these gaps, we contrast zero-shot prediction (ZSP) with the related setting of few-shot learning (FSL). Let x X denote an input (often an image) that accompanies a discrete value y Y (often a class label).

Improving the reasoning capabilities of diffusion-based large language models (dLLMs) through reinforcement learning (RL) remains an open problem. The intractability of dLLMs likelihood function necessitates approximating the current, old, and reference policy likelihoods at each policy optimization step. This reliance introduces additional computational overhead and lead to potentially large bias -- particularly when approximation errors occur in the denominator of policy ratios used for importance sampling. To mitigate these issues, we introduce $\mathtt{wd1}$, a novel policy optimization approach that reformulates the objective as a weighted likelihood, requiring only a single approximation for the current parametrized policy likelihood. Experiments on widely used reasoning benchmarks demonstrate that $\mathtt{wd1}$, without supervised fine-tuning (SFT) or any supervised data, outperforms existing RL methods for dLLMs, achieving up to 16% higher accuracy. $\mathtt{wd1}$ delivers additional computational gains, including reduced training time and fewer function evaluations (NFEs) per gradient step. These findings, combined with the simplicity of method's implementation and R1-Zero-like training (no SFT), position $\mathtt{wd1}$ as a more effective and efficient method for applying RL to dLLMs reasoning.

The zero-shot capabilities of foundation models (FMs) for time series forecasting offer promising potentials in conformal prediction, as most of the available data can be allocated to calibration. This study compares the performance of Time Series Foundation Models (TSFMs) with traditional methods, including statistical models and gradient boosting, within a conformal prediction setting. Our findings highlight two key advantages of TSFMs. First, when the volume of data is limited, TSFMs provide more reliable conformalized prediction intervals than classic models, thanks to their superior predictive accuracy. Second, the calibration process is more stable because more data are used for calibration. Morever, the fewer data available, the more pronounced these benefits become, as classic models require a substantial amount of data for effective training. These results underscore the potential of foundation models in improving conformal prediction reliability in time series applications, particularly in data-constrained cases. All the code to reproduce the experiments is available on GitHub.

We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference time. Traditional language modeling is an intrinsically ill-posed problem: given a context, multiple futures may be equally plausible. Our approach leverages Multiple Choice Learning (MCL) and the Winner-Takes-All (WTA) loss to efficiently handle ambiguity through Low-Rank Adaptation (LoRA). We provide a theoretical interpretation of applying Multiple Choice Learning to Language Modeling, assuming the data is generated from a mixture of distributions. To illustrate the proposed approach, we use data sampled from mixtures of Markov chains. We then demonstrate with extensive experiments on real-world visual and audio captioning tasks that our method achieves high diversity and relevance in generated outputs.