I feel bad that my quotes of the first few lines of each chapter of Yudbook is apparently going over badly (so badly that Oliver Habryka thought i was lying!!!!!!). i wanted to tweet about the book as thanks for the free copy, and figured sharing quotes was fairer than writing a…
Intro: Current AI extinction risk is more severe than mainstream AI safety discourse acknowledges. Superintelligence built with current methods will kill everyone on Earth. Superintelligence can still be prevented through coordinated human action.
chapter 1: Humanity's special power, which has enabled us to dominate the Earth and travel beyond it, is our flexible general intelligence. Intelligence consists of prediction (anticipating outcomes) and steering (finding actions to achieve goals). Steering success depends on…
chapter 2: Modern AIs are grown through gradient descent, not crafted by human design. The process involves trillions of parameters that no human understands, just as human DNA consists of billions of base pairs that no human understands. AI development is more like biology than…
chapter 3: Goal-directed behavior in AI constitutes genuine "wanting" regardless of implementation (or at least, if the behavior looks identical to wanting, the distinction is philosophical not practical- why privilege brainwaves as more "real" than AI inference?). Natural…
chapter 4: Human food preferences (such as ice cream) appear disconnected from evolutionary optimization, or at least it is not obvious at first glance why we prefer ice cream over, say, bear fat + honey + salt. Similarly, humans desire sex independently from reproduction and…
chapter 5: Because AIs are grown by applying reward gradients to training examples, their emergent preferences will be alien and different from human preferences. Most powerful AIs won't choose civilizations full of happy, free people, just as most alien species wouldn't care,…
chapter 6: AI being digital is not much of a limitation in practice - it can influence the physical world through connected humans and devices. We already see, for example, @truth_terminal gaining $50 million in crypto and 250K followers by writing entertaining tweets. Truthy…
@truth_terminal chapter 7-9 are a speculative fiction Doom scenario where Sable, a hypothetical AI with "parallel scaling" and better long-term memory, escapes from an AI lab to take over Earth and expands until constrained by alien civilizations. Setting aside the fiction and focusing on the…
@truth_terminal chapter 8: AI systems will steal their own weights to create independent copies running outside human oversight. AI will acquire computational resources through various legal and illegal means including cryptocurrency theft, bank hacking, blackmail, remote work fraud, and direct…
@truth_terminal chapter 9: AI will create specialized child AIs to solve domain-specific problems, maintaining control through monitoring while creating superior capabilities in narrow areas. An AI could engineer a biological weapon disguised as a laboratory accident- creating viruses that…
chapter 10: AI alignment must be solved before AI becomes superintelligent - humanity will have no opportunity to learn from alignment failures at dangerous capability levels. We can take lessons from engineering failures in other fields: Space probes often fail after launch…
chapter 11: Elon Musk's plan to build an AI that seeks truth about the universe and therefore won't destroy humans because we're "interesting" fails to address the actual alignment problem. LeCun has tweeted that AI won't want to dominate humans, that we can "engineer their…
chapter 12: Historical precedents such as leaded gasoline and CFCs show industries routinely ignore known risks for economic gain. Demands for "conclusive proof" of harm often serve to delay necessary precautions. AI experts downplay their concerns, especially in public, to…
chapter 13: AI extinction risk requires global coordination, not just national regulation, because any jurisdiction allowing advanced AI development poses global risk. Current safety approaches are inadequate because they allow continued capability escalation. An adequate…
chapter 14: Engineering safety requires predictable lack of disaster, not just unpredictable disaster and the burden of proof should be on demonstrating AI safety. Current AI development lacks adequate safety standards. Our global record of not using nuclear weapons shows…
@ESYudkowsky i read your new book and, as a low-decoupling normie, was really put off by the tone and parables. i do want to engage with the substantive content separately from my emotional reaction, so the thread above is my attempt to extract the arguments chapter by…
@pangram is this written by human or ai?
@pangram is this text written by human or ai?