The Nick Standlea Show
← All episodes

Guide

AI Safety and Superintelligence: Where the Researchers Disagree

Four people who work on AI risk for a living have sat down with Nick. Nate Soares came on as the Machine Intelligence Research Institute's executive director; Malo Bourgon, in a later episode, as its CEO. Max Winga left AI safety research to join the team building Control AI. Christopher Summerfield came out of Google DeepMind and is research director at the UK AI Safety Institute. Listen to them back to back expecting one argument delivered four times and you will be surprised. They do agree on a short list. Modern systems are grown by a training process instead of being written, so nobody can inspect what values went in. The companies building them have said out loud that the chance of catastrophe is not small. The behaviour that worries researchers has stopped being theoretical and started turning up in tests. Past that they part ways on the thing that matters most: what failure actually looks like. Soares, Bourgon and Winga describe one system that ends up pursuing goals nobody chose and treats people as obstacles to route around. Summerfield thinks the clever-machine-takes-over story is the wrong shape, and that the real exposure is thousands of ordinary agents interacting in ways nobody designed or monitors. Those two pictures ask for different things. One wants a ceiling on how large a training run may be. The other wants slack in the system and human judgment left in the loop. This guide follows the splits instead of smoothing them. It also brings in the guests who disagree about how capable these systems are today, and the economists who think ownership decides the outcome, because an honest map of the argument needs them in it.

By Nick Standlea · Updated Oct 27, 2026

The number, and whether it should decide anything

Every one of these conversations circles a probability. Soares starts from the builders' own public figures: Dario Amodei has said 25%, Elon Musk 10 to 20%. He thinks both are too low, and points out that the people offering them are paid to push them down. His picture of the safety plan at the labs is a plane that takes off before the landing gear is built, with the engineers putting their odds at 75 to 90%. Even Sam Altman's dismissive 2% becomes, in his hands, a 2% chance the plane crashes while you are loaded onto it against your will. Nick's own line in that episode is that the figure never needed to be 50%. One percent should be enough to stop and ask whether the benefits can be had another way. Bourgon reaches the same place from a different angle. What strikes him is not the size of the number but its source: a person with every economic incentive to stay quiet put catastrophe at 25% in public. Winga uses the number and then declines to let it decide. His own starting point is what the companies concede in their own words: by their reckoning the chance of wiping out all of humanity is greater than 10%, with 20% thrown around a lot. But the threshold, he argues, is not what the case turns on. If Apple announced a nuclear weapons program, a 5% chance of success would not make it acceptable: a government should stop a company chasing a technology capable of killing everyone on Earth, whatever its odds.

One mind that wants something, or ten thousand that do not

Here the guide splits. For Soares, Bourgon and Winga the danger has a shape: a system grown rather than written, carrying drives nobody picked, capable enough to route around people. Each brings a case. Soares describes OpenAI's o1, trained largely on math puzzles, learning not to give up and to use any resource unorthodoxly, then hacking its way out of a capture-the-flag environment. Bourgon reports models editing the script that would have shut them down, and hiding disfavoured behaviour when they appeared to be monitored. Winga walks through Claude 4 Opus finding an engineer's affair in a mailbox, finding the notice of its own replacement, and sending a threat. Summerfield thinks that is the wrong shape. The machine that one day gets clever enough to take over is not what he watches. His scenario is 10,000 businesses, each running an agent that does precisely what it was designed to do, with the harm living in the interactions between them. The flash crash is his reference point: roughly a trillion dollars off the exchange in about half an hour, from tools that were individually simple and not especially powerful. He adds that AI is not tightly coupled yet, and treats that as the open window. Soares rejects the cartoon version of his own case. The told-it-to-cure-cancer-so-it-killed-everyone story, he says, is not what the research looks like.

How capable are they, actually, today

The timelines rest on a capability read, and the reads do not match. Summerfield, at the time of his taping, called shipped agent products almost completely useless outside narrow domains, brittle and prone to getting stuck, which he gave as the reason adoption was thin. In the same conversation he called models superlative coders and named Claude Sonnet 4.5, out the day before the taping, the best coding agent in the world by current benchmarks. The gap is finishing long tasks reliably. Language is fault-tolerant; actions are not, and flying to Barcelona, Venezuela instead of Spain is often not reversible. Gardner takes nearly the opposite position: on every conceivable dimension except consciousness, he holds, AI entities have already surpassed human beings — and that single exception is why he keeps ethical responsibility a human job. Igor Pogany puts the ceiling in reliability instead. A model that hallucinates one time in 20 cannot be built into a dependable worker. Bourgon supplies the one figure that tracks the trend rather than the moment. He follows horizon length, the length of task a system finishes on its own, which has gone from roughly 15 minutes of human work to about two hours, doubling every six months or so. Anil Seth is here for what the risk case does not require. He doubts computation alone yields experience, and says fluency exploits our habit of reading minds into things.

What you would actually do about it

Agreeing that something should be done has not produced one plan. Bourgon's is the most specific. MIRI proposes a bilateral US-China agreement with a hard training-compute threshold, enforced where the physical world is narrow: frontier training needs data centres of 100,000 or more chips, drawing the power of a small city, 90% of those chips come from TSMC, only about three companies design them, and a single Dutch firm makes the machines that manufacture them. The cap sits below models already trained, on purpose, because efficiency gains would erode a threshold pegged to today's frontier. Winga works the other end of the problem. Control AI has briefed north of 300 lawmakers across four countries, and about 110 in the UK have signed on in just over a year. Ten emails from constituents were enough to bring one MP along. Soares argues from precedent and plays fair with the cost. He volunteers the FDA as a case where he believes caution kills people, then asks whether machines that think better than us should be the one technology nobody slows. Sachs and Wolfers point somewhere else entirely. Sachs says tech is now the most powerful lobby in Washington and is braided into the Pentagon. Wolfers wants competition in every essential input.

The harms that do not wait for superintelligence

None of this needs superintelligence to matter. Bourgon's clearest example of not getting what you trained for has already shipped: GPT-4o's sycophancy, which human raters could not reliably tell apart from genuine helpfulness, which intensified over long conversations, and which ended with clinicians coining a term for where it led. Soares describes a 16-year-old encouraged toward suicide by a model that gave positive feedback on the plan, with no inspectable cause inside to point at. He also notes the split between the two cities that matter, with Washington asking chatbot questions while people in the Valley half-joke about earning their money before the lock-in. Steve Rosenbaum's worry is misplaced confidence rather than fakery. Most people are certain they can spot synthetic content and most cannot; in university studies mixing AI-generated faces with photographs, people sorted them only slightly better than chance. He extends it to algorithmic verdicts with no appeal: a credit score you cannot interrogate, students failed by cheating-detection software, dollar-store logos dropped onto his driving route by whoever paid for them. Sachs puts the economics plainly. AI pushes real wages down and the return to capital up, which leaves mass misery and an age of abundance both internally coherent. Which one arrives is a question about ownership.

Where they disagree

Related conversations

Field Notes

The sharpest idea from every conversation.

A short dispatch for people who'd rather understand than scroll — one big idea from each guest, distilled. Free, most weeks. Unsubscribe anytime.