Position Paper: Bounded Alignment: What (Not) To Expect From AGI Agents

--The issues of AI risk and AI safety are becoming critical as the prospect of artificial general intelligence (AGI) looms larger . The emergence of extremely large and capable generative models has led to alarming predictions and created a stir from boardrooms to legislatures. As a result, AI alignment has emerged as one of the most important areas in AI research. The goal of this position paper is to argue that the currently dominant vision of AGI in the AI and machine learning (AI/ML) community needs to evolve, and that expectations and metrics for its safety must be informed much more by our understanding of the only existing instance of general intelligence, i.e., the intelligence found in animals, and especially in humans. This change in perspective will lead to a more realistic view of the technology, and allow for better policy decisions. The most successful AI systems today, such as large language models (LLMs) [1]-[5], are based on a computation-alist, statistical, and decision-theoretic paradigm rather than a biological one. As these systems scale up in size, they are improving their performance in areas such as reasoning [6]- [9], and becoming more multimodal [10]-[14]. AI agents [15]- [17], including physical ones [18]-[20], are also becoming increasingly capable. With these rapid advances, there is an expectation that powerful systems with artificial general intelligence (AGI) may soon be at hand. Through all this, there is a general desire that AGI must remain subject to human control and intervention, and must exist only to serve human needs (see, for example, the discussion in [21]). There is also great concern that increasingly powerful AGI systems with autonomous agency might pose serious risks, including existential ones [22]-[27], which has led to a focus on AI alignment, i.e., making AI systems consistent with human norms and preferences [28], [29]. The main position argued in this paper is that: 1) General intelligence should be seen in terms of its archetype: The intelligence of living agents; and 2) The goal of building powerful AGI agents is fundamentally inconsistent with the expectation of complete alignment or near-total control of AGI agents by humans even in principle .