@ESYudkowsky Why should we believe that foom of exponential self-improvement would actually happen in practice? In more detail than "evolutionary pressure" and "Moores Law" Also why would ASI scale up without limits as a single agent w internal alignment?
More notes on nearby ideas
Reply · Feb 16, 2023@nobuhojimichaan foom is when AGI is able to self-improve exponentially and suddenly become superintelligence... doom is negative-to-apocalyptic takes…Reply · Aug 29, 2023@samswoora yeah I kinda think this is an important intuition- we have a lot of human-level examples to learn from, not a lot of super-human... tho I'm…Reply · Apr 19, 2024@RokoMijic @ESYudkowsky @robinhanson i remember debating w you last year about infrastructure & takeoff speed - good for you to update on evidence! 🫶…Tweet · Mar 28, 2024sometimes i kinda wonder if exponentially self improving AGI is the new perpetual motion machine or faster than light travel