The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that policies are treated as being separate from the world they inhabit. This leads to theoretical challenges in the multi-agent setting where the non-stationarity induced by the learning of other agents demands prospective learning based on prediction models. To accurately model other agents, an agent must account for the fact that those other agents are, in turn, forming beliefs about it to predict its future behavior, motivating agents to model themselves as part of the environment. Here, building upon foundational work on universal artificial intelligence (AIXI), we introduce a mathematical framework for prospective learning and embedded agency centered on self-prediction, where Bayesian RL agents predict both future perceptual inputs and their own actions, and must therefore resolve epistemic uncertainty about themselves as part of the universe they inhabit. We show that in multi-agent settings, self-prediction enables agents to reason about others running similar algorithms, leading to new game-theoretic solution concepts and novel forms of cooperation unattainable by classical decoupled agents. Moreover, we extend the theory of AIXI, and study universally intelligent embedded agents which start from a Solomonoff prior. We show that these idealized agents can form consistent mutual predictions and achieve infinite-order theory of mind, potentially setting a gold standard for embedded multi-agent learning.

A Bayesian player acting in an infinite multi-player game learns to predict the other players' strategies if his prior assigns positive probability to their play (or contains a grain of truth). Kalai and Lehrer's classic grain of truth problem is to find a reasonably large class of strategies that contains the Bayes-optimal policies with respect to this class, allowing mutually-consistent beliefs about strategy choice that obey the rules of Bayesian inference. Only small classes are known to have a grain of truth and the literature contains several related impossibility results. In this paper we present a formal and general solution to the full grain of truth problem: we construct a class of strategies wide enough to contain all computable strategies as well as Bayes-optimal strategies for every reasonable prior over the class. When the "environment" is a known repeated stage game, we show convergence in the sense of [KL93a] and [KL93b]. When the environment is unknown, agents using Thompson sampling converge to play $\varepsilon$-Nash equilibria in arbitrary unknown computable multi-agent environments. Finally, we include an application to self-predictive policies that avoid planning. While these results use computability theory only as a conceptual tool to solve a classic game theory problem, we show that our solution can naturally be computationally approximated arbitrarily closely.

The original AIXI reinforcement learning agent, intended as a near ly parameter-free formal gold standard for artificial general intelligence (AGI), is a Cartesian dualist that believes it is interacting with an environment from the outside, in the sense that its policy is fixed and not overwritten by anything that happens in the environment, though its actions can certainly adapt based on the percepts it receives. This is frequently compared to a person playin g a video game, who certainly does not believe he is being simulated by the game b ut rather interacts with it only by observing the screen and pressing b uttons. In contrast, it would presumably be important for an AGI to be aware t hat it exists within its environment (the universe) and its computations ar e therefore subject to the laws of physics. With this in mind, we investigate versio ns of the AIXI agent [Hut00] that treat the action sequence a on a similar footing to the percept sequence e, meaning that the actions are considered as explainable by the same rules generating the percepts. The most obvious idea is to use the universal distribution to model the joint (action/percept) dis tribution (even though actions are selected by the agent). Although this is the mos t direct way to transform AIXI into an embedded agent, it does not appear to h ave been analyzed in detail; in particular, it is usually assumed (but not proven) to fail (often implicitly, without distinguishing the universal sequence and environment distributions, e.g.

Perhaps the most impressive feature of today's frontier models is their ability to swiftly adapt their behavior to a wide range of contexts. Given relatively few tokens--whether from a user input, a system prompt, or a number of in-context examples--models often rapidly infer the task at hand and produce good continuations without any weight adaptation (in-context learning, Lampinen et al. [2024]). From a meta-learning perspective, rapid in-context adaptation is expected to arise: log loss minimization with a parametric sequential predictor (like a neural network) over a distribution of stochastic data generators leads to a Bayesian predictor for the pretraining distribution [Ortega et al., 2019]. The hallmark feature of such a predictor (Bayes-optimality) is most rapid in-context adaptation and least (cumulative) prediction error on average. Accordingly, prompting, that is conditioning of the Bayesian predictor, can be used to data-efficiently adapt the pretrained model to a target task. An important question is: under which conditions is it possible to find a prompt such that the prompted pretrained predictor becomes (near-) Bayes-optimal on a target task? We refer to this as optimal prompting, which is possible in theory if the target task is one of the tasks covered by the meta-distribution. If this is not the case, then optimal prompting may not be possible for an ideal predictor, and weight adaptation may be necessary.

In contrast to popular decision-making frameworks such as reinforcement learning, which are built upon appealing to decision-theoretic notions such as Maximum Expected Utility, we instead construct an agent by trying to minimise the expected number of bits needed to losslessly describe general agent-environment interactions. The appeal with this approach is that if we can construct a good universal coding scheme for arbitrary agent interactions, one could simply sample from this coding distribution to generate a control policy. However when considering general agents, whose goal is to work well across multiple environments, this question turns out to be surprisingly subtle. Naive approaches which do not discriminate between actions and observations fail, and are subject to the self-delusion problem [Ortega et al., 2021]. In this work, we will adopt a universal source coding perspective to this question, and showcase its efficacy by applying it to the challenging non-stationary stochastic bandit problem. In the passive case, namely, sequential prediction of observations under the logarithmic loss, there is a well developed universal source coding literature for dealing with non-stationary sources under various types of non-stationarity. The most influential idea for modelling piecewise stationary sources is the transition diagram technique of Willems [1996]. This technique performs Bayesian model averaging over all possible partitions of a sequence of data, cleverly exploiting dynamic programming to yield an algorithm which has both quadratic time complexity and provable regret guarantees.

Large language models (LLMs) can be prompted to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We explore these issues by viewing prompting as conditioning a near-optimal sequence predictor (LLM) pretrained on diverse data sources. Through numerous prompt search experiments, we show that the unintuitive patterns in optimal prompts can be better understood given the pretraining distribution, which is often unavailable in practice. Moreover, even using exhaustive search, reliably identifying optimal prompts from practical neural predictors can be difficult. Further, we demonstrate that common prompting methods, such as using intuitive prompts or samples from the targeted task, are in fact suboptimal. Thus, this work takes an initial step towards understanding the difficulties in finding and understanding optimal prompts from a statistical and empirical perspective.

We are interested in tree search algorithms for deterministic domains. Tree search algorithms such as all variants of best-first search -- including A* [Hart et al., 1968], Weighted-A* (WA*), and Greedy Best-First Search (GBFS) [Doran et al., 1966] -- and variants of MCTS -- such as UCT [Kocsis and Szepesvári, 2006], AlphaGo, AlphaZero and other variants [Silver et al., 2016, 2017b,a] -- explore the search tree starting at the root and can visit a node only if its parent has been visited first. These algorithms are often guided by some side information, such as with cost-to-go heuristic function for A*, WA* and GBFS, a reward/value function for UCT and AlphaZero, or a policy for AlphaZero, Levin Tree Search (LTS) [Orseau et al., 2018], and Policy-Guided Heuristic Search [Orseau and Lelis, 2021]. Such algorithms also sometimes come with different types of guarantees: A* and WA*, with an admissible heuristic function -- i.e., a function that never overestimates the optimal costto-go -- are guaranteed to return a solution that is cost-optimal (A*) or bounded-suboptimal (WA*), while UCT and AlphaZero are guaranteed to (eventually) have low regret in terms of cumulative reward during the search. LTS is guaranteed to return a solution within a number of search steps that depends on the quality of its policy. In this paper, we consider the latter type of guarantee, on the efficiency of the search process depending on the quality of the side information. To explain the main concepts of this paper, let us consider a kind of side information we call clues: some nodes are clue nodes, and a node can be known to be a clue node only when reaching it. A clue may be helpful if it is on the path toward a solution node, or misleading otherwise. The following example describes a minimalistic clue environment.

In reinforcement learning, if the agent's reward differs from the designers' true utility, even only rarely, the state distribution resulting from the agent's policy can be very bad, in theory and in practice. When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do"). All current cutting-edge language models are RL agents that are KL-regularized to a "base policy" that is purely predictive. Unfortunately, we demonstrate that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We demonstrate this theoretically using algorithmic information theory, and while systems today are too weak to exhibit this theorized failure precisely, we RL-finetune a language model and find evidence that our formal results are plausibly relevant in practice. We also propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do".

To understand the risks posed by a new AI system, we must understand what it can and cannot do. Building on prior work, we introduce a programme of new "dangerous capability" evaluations and pilot them on Gemini 1.0 models. Our evaluations cover four areas: (1) persuasion and deception; (2) cyber-security; (3) self-proliferation; and (4) self-reasoning. We do not find evidence of strong dangerous capabilities in the models we evaluated, but we flag early warning signs. Our goal is to help advance a rigorous science of dangerous capability evaluation, in preparation for future models.

We consider the problem of online fine tuning the parameters of a language model at test time, also known as dynamic evaluation. While it is generally known that this approach improves the overall predictive performance, especially when considering distributional shift between training and evaluation data, we here emphasize the perspective that online adaptation turns parameters into temporally changing states and provides a form of context-length extension with memory in weights, more in line with the concept of memory in neuroscience. We pay particular attention to the speed of adaptation (in terms of sample efficiency),sensitivity to the overall distributional drift, and the computational overhead for performing gradient computations and parameter updates. Our empirical study provides insights on when online adaptation is particularly interesting. We highlight that with online adaptation the conceptual distinction between in-context learning and fine tuning blurs: both are methods to condition the model on previously observed tokens.