Formalizing Embeddedness Failures in Universal Artificial Intelligence

The original AIXI reinforcement learning agent, intended as a near ly parameter-free formal gold standard for artificial general intelligence (AGI), is a Cartesian dualist that believes it is interacting with an environment from the outside, in the sense that its policy is fixed and not overwritten by anything that happens in the environment, though its actions can certainly adapt based on the percepts it receives. This is frequently compared to a person playin g a video game, who certainly does not believe he is being simulated by the game b ut rather interacts with it only by observing the screen and pressing b uttons. In contrast, it would presumably be important for an AGI to be aware t hat it exists within its environment (the universe) and its computations ar e therefore subject to the laws of physics. With this in mind, we investigate versio ns of the AIXI agent [Hut00] that treat the action sequence a on a similar footing to the percept sequence e, meaning that the actions are considered as explainable by the same rules generating the percepts. The most obvious idea is to use the universal distribution to model the joint (action/percept) dis tribution (even though actions are selected by the agent). Although this is the mos t direct way to transform AIXI into an embedded agent, it does not appear to h ave been analyzed in detail; in particular, it is usually assumed (but not proven) to fail (often implicitly, without distinguishing the universal sequence and environment distributions, e.g.