Nate Soares on P(Doom), Alien Superintelligence, Human Enhancement, and the Future of AI

Why did one of the world’s earliest and most influential AI alignment research organizations fail to solve the problem it was created to address? In this episode of The Roman Forum, Dr. Roman Yampolskiy speaks with Nate Soares, president of the Machine Intelligence Research Institute and co-author of If Anyone Builds It, Everyone Dies, about the unresolved problem of aligning superintelligent AI with human values. Soares argues that alignment may be physically possible in principle, yet remains far beyond the reach of current methods. Humanity may need centuries of intellectual progress to solve it, while advanced AI could arrive within years. The conversation examines why MIRI’s technical research did not produce a workable solution, whether an AI can genuinely care about human flourishing, and why ordinary scientific trial and error may be impossible when a single failed experiment could become irreversible. They also discuss coherent extrapolated volition, human value aggregation, the distinction between alignment and control, the point of no return, alien superintelligence, the Fermi paradox, acausal trade, P(doom), human cognitive enhancement, and the possibility of enforcing a global pause on frontier AI development. Soares’s central proposal is direct: humanity should stop the race toward superintelligence before systems become capable of automating AI research and accelerating beyond meaningful human control. He explains why a pause would need to be international, how advanced chips and large data centers could be monitored, and why the world may need to treat superintelligence development with a seriousness comparable to nuclear proliferation. http://so8res.com/ 00:00 Why MIRI Failed to Solve AI Alignment 02:34 What the Alignment Problem Really Is 05:25 Can AI Learn What Humanity Truly Wants? 07:40 Could “Aligned” AI Decide to Kill Everyone? 12:55 Whose Values Should Superintelligence Follow? 19:03 Alignment vs. Control 21:08 The Point of No Return 23:44 Alien Superintelligence and the Fermi Paradox 30:04 Could AI Sell Humanity to Aliens? 37:51 Do We Have 12 Months or 12 Years? 39:01 The Proposal: Shut It All Down 46:14 How Do We Legally Ban Superintelligence? 55:33 Nate Soares on P(Doom) 01:00:31 Could a Global AI Pause Actually Work? 01:12:34 Nuclear War or Uncontrolled Superintelligence? 01:19:29 How Close Are We to Political Action? 01:26:09 The Best Arguments Against AI Doom Who is Roman Yampolskiy: https://grokipedia.com/page/roman_yam... Research papers: https://scholar.google.com/citations?... Books: AI: Unexplainable, Unpredictable, Uncontrollable https://www.amazon.com/Unexplainable-... Considerations on the AI Endgame https://www.amazon.com/Considerations... Artificial Superintelligence: A Futuristic Approach https://www.amazon.com/Artificial-Sup... Artificial Intelligence Safety and Security https://www.amazon.com/Artificial-Int... Social Media X https://x.com/romanyam FB   / roman.yampolskiy   IN   / romanyam   Ask Roman to speak at your event: https://www.romanyampolskiy.com/