chapter 3: Goal-directed behavior in AI constitutes genuine "wanting" regardless of implementation (or at least, if the behavior looks identical to wanting, the distinction is philosophical not practical- why privilege brainwaves as more "real" than AI inference?). Natural…
chapter 4: Human food preferences (such as ice cream) appear disconnected from evolutionary optimization, or at least it is not obvious at first glance why we prefer ice cream over, say, bear fat + honey + salt. Similarly, humans desire sex independently from reproduction and…
chapter 5: Because AIs are grown by applying reward gradients to training examples, their emergent preferences will be alien and different from human preferences. Most powerful AIs won't choose civilizations full of happy, free people, just as most alien species wouldn't care,…