existential risk
Timnit Gebru Believes There Is No 'Existential Threat' From AI
Timnit Gebru Believes There Is No'Existential Threat' From AI One of AI's fiercest critics believes the doom talk is about founders making money, not saving humanity. As the industry booms, an entirely new vernacular has emerged in the debates over how consequential artificial intelligence really is. Within that new parlance, two phrases have become focal points: stochastic parrots and existential risk . And at the center of is one prominent AI researcher who has never stood down from a fight, Timnit Gebru . Gebru came into the public eye several years ago after sparring with Google over a research paper she coauthored that called out biases in the company's AI, saying that LLMs basically parroted their training data and risked perpetuating biased viewpoints. The contested paper resulted in Gebru's departure from the company, and led her to found an institute that investigates harms perpetuated by technology and supports the creation of unbiased tech tools. She has also authored a new book,, expected to ship early next year. More recently, Gebru has spoken out against a faction of the industry that believes AI is so powerful it could destroy humanity. In turn, some members of that community, including an Anthropic cofounder, have alleged that Gebru's earlier research about stochastic parrots is no longer relevant, and that AI does have the ability to "think" or reason. The AI debate is no longer just about the technology itself, but about ideological groups and strategic narratives, and Gebru believes these narratives are a "harmful distraction" from the real issues with AI. I recently spoke with Gebru about what she believes these arguments are distracting from, and also got her response to critiques that her earlier research underestimates the AI of today.
Why China Isn't Getting Existential About A.I.
China's fears about the technology are increasingly distinct from those of the U.S. What does that mean for international coรถperation? On Sunday, China's Minister of State Security published an article in a state-run magazine that warned about artificial intelligence posing a threat to the rule of the Chinese Communist Party (C.C.P.). This coincided with increasing concern among Americans that A.I. could pose existential risks to humanity, and rising calls for Washington to regulate it. Many of those pushing for regulation work at leading A.I. companies such as Anthropic and OpenAI, and they have received support from a broad ideological coalition on Capitol Hill. But there remains little appetite for regulation in the White House; President Trump has said that the United States must accelerate A.I. development to outpace China. I recently spoke by phone with Kyle Chan, a fellow at the Brookings Institution who focusses on Chinese technology policy.
How AI firm Anthropic wound up in the Pentagon's crosshairs
This week has brought more chaos in the feud between the Pentagon and Anthropic. This week has brought more chaos in the feud between the Pentagon and Anthropic. How AI firm Anthropic wound up in the Pentagon's crosshairs U ntil recently, Anthropic was one of the quieter names in the artificial intelligence boom. Despite being valued at about $350bn, it rarely generated the flashy headlines or public backlash associated with Sam Altman's OpenAI or Elon Musk's xAI. Its CEO and co-founder Dario Amodei was an industry fixture but hardly a household name outside of Silicon Valley, and its chatbot Claude lagged in popularity behind ChatGPT.
AI Consciousness and Existential Risk
In AI, the existential risk denotes the hypothetical threat posed by an artificial system that would possess both the capability and the objective, either directly or indirectly, to eradicate humanity. This issue is gaining prominence in scientific debate due to recent technical advancements and increased media coverage. In parallel, AI progress has sparked speculation and studies about the potential emergence of artificial consciousness. The two questions, AI consciousness and existential risk, are sometimes conflated, as if the former entailed the latter. Here, I explain that this view stems from a common confusion between consciousness and intelligence. Yet these two properties are empirically and theoretically distinct. Arguably, while intelligence is a direct predictor of an AI system's existential threat, consciousness is not. There are, however, certain incidental scenarios in which consciousness could influence existential risk, in either direction. Consciousness could be viewed as a means towards AI alignment, thereby lowering existential risk; or, it could be a precondition for reaching certain capabilities or levels of intelligence, and thus positively related to existential risk. Recognizing these distinctions can help AI safety researchers and public policymakers focus on the most pressing issues.
AI Safety Should Prioritize the Future of Work
Hazra, Sanchaita, Majumder, Bodhisattwa Prasad, Chakrabarty, Tuhin
Current efforts in AI safety prioritize filtering harmful content, preventing manipulation of human behavior, and eliminating existential risks in cybersecurity or biosecurity. While pressing, this narrow focus overlooks critical human-centric considerations that shape the long-term trajectory of a society. In this position paper, we identify the risks of overlooking the impact of AI on the future of work and recommend comprehensive transition support towards the evolution of meaningful labor with human agency. Through the lens of economic theories, we highlight the intertemporal impacts of AI on human livelihood and the structural changes in labor markets that exacerbate income inequality. Additionally, the closed-source approach of major stakeholders in AI development resembles rent-seeking behavior through exploiting resources, breeding mediocrity in creative labor, and monopolizing innovation. To address this, we argue in favor of a robust international copyright anatomy supported by implementing collective licensing that ensures fair compensation mechanisms for using data to train AI models. We strongly recommend a pro-worker framework of global AI governance to enhance shared prosperity and economic justice while reducing technical debt.
Mitigating Societal Cognitive Overload in the Age of AI: Challenges and Directions
Societal cognitive overload, driven by the deluge of inform ation and complexity in the AI age, poses a critical challenge to human well-being an d societal resilience. This paper argues that mitigating cognitive overload is not only essential for improving present-day life but also a crucial prerequisite fo r navigating the potential risks of advanced AI, including existential threats. W e exa mine how AI exacerbates cognitive overload through various mechanisms, incl uding information proliferation, algorithmic manipulation, automation anxiet ies, deregulation, and the erosion of meaning. The paper reframes the AI safety debate t o center on cognitive overload, highlighting its role as a bridge between near-te rm harms and long-term risks. It concludes by discussing potential institutional adaptations, research directions, and policy considerations that arise from adopti ng an overload-resilient perspective on human-AI alignment, suggesting pathways fo r future exploration rather than prescribing definitive solutions. W e stand at a precipice. Human societies are increasingly st ruggling to process the sheer volume and complexity of information in the digital age, a conditio n dramatically amplified by the rapid proliferation of artificial intelligence (AI). While Toffle r (1970) foresaw "future shock" from accelerating change and Eppler & Mengis (2004); Bawden & Robin son (2009) analyzed individual information overload, Byung-Chul Han, in his critique of ne oliberalism and technological domination (Han, 2017), argues that contemporary society faces a regime of technological domination that exploits and overwhelms the psyche. This exploitation and overwhelming of the psyche, now dramatically amplified by AI-driven information and comple xity, elevates information overload to a systemic crisis: societal cognitive overload .
When Autonomy Breaks: The Hidden Existential Risk of AI
AI risks are typically framed around physical threats to humanity, a loss of control or an accidental error causing humanity's extinction. However, I argue in line with the gradual disempowerment thesis, that there is an underappreciated risk in the slow and irrevocable decline of human autonomy. As AI starts to outcompete humans in various areas of life, a tipping point will be reached where it no longer makes sense to rely on human decision-making, creativity, social care or even leadership. What may follow is a process of gradual de-skilling, where we lose skills that we currently take for granted. Traditionally, it is argued that AI will gain human skills over time, and that these skills are innate and immutable in humans. By contrast, I argue that humans may lose such skills as critical thinking, decision-making and even social care in an AGI world. The biggest threat to humanity is therefore not that machines will become more like humans, but that humans will become more like machines.
The Economics of p(doom): Scenarios of Existential Risk and Economic Growth in the Age of Transformative AI
Growiec, Jakub, Prettner, Klaus
Recent advances in artificial intelligence (AI) have led to a diverse set of predictions about its long-term impact on humanity. A central focus is the potential emergence of transformative AI (TAI), eventually capable of outperforming humans in all economically valuable tasks and fully automating labor. Discussed scenarios range from human extinction after a misaligned TAI takes over ("AI doom") to unprecedented economic growth and abundance ("post-scarcity"). However, the probabilities and implications of these scenarios remain highly uncertain. Here, we organize the various scenarios and evaluate their associated existential risks and economic outcomes in terms of aggregate welfare. Our analysis shows that even low-probability catastrophic outcomes justify large investments in AI safety and alignment research. We find that the optimizing representative individual would rationally allocate substantial resources to mitigate extinction risk; in some cases, she would prefer not to develop TAI at all. This result highlights that current global efforts in AI safety and alignment research are vastly insufficient relative to the scale and urgency of existential risks posed by TAI. Our findings therefore underscore the need for stronger safeguards to balance the potential economic benefits of TAI with the prevention of irreversible harm. Addressing these risks is crucial for steering technological progress toward sustainable human prosperity.