Can we keep control of AI? An Oxford neuroscientist answers

He is a professor of cognitive neuroscience at the University of Oxford and the research director at the UK AI Safety Institute. He was formerly a senior research scientist at Google DeepMind. He wrote These Strange New Minds, a book about how large language models learned to talk and how they compare to the human brain.

Nick opened with that exact question. Summerfield does not think the fast takeoff story, where a system wakes up one day clever enough to take over the world, is very likely. What concerns him is AI getting built into infrastructure, politics, business, finance, and telecoms until consequential decisions are being made for us. In his words, the main story around AI taking over is really a story about control.

Summerfield sorts risk into three buckets. There is deliberate misuse by a human actor who wants to cause harm. There are inadvertent errors, where the model is trying to help but doesn't understand the norms, laws, or context well enough. And there are systemic or secondary effects, which he thinks get talked about least and may matter most.

Psychology, economics, and machine learning all landed on roughly the same answer about what people want, which is reward. Summerfield says that story doesn't hold up against child development, evolution, or how people actually spend their time. Marathon runners are his example: months of punishing training for something that isn't comfort or money. What we're really after is the ability to influence the world in predictable ways.

Summerfield describes an agent as an extension of a generative AI system like ChatGPT, Gemini, or Claude. Instead of only putting out text, it puts out special tokens that follow protocols and get sent on to browsers, datasets, content management systems, and other applications. So rather than telling you Paris is the capital of France, it can carry an instruction like buy this product for me.

Summerfield says the products on public release are almost completely useless outside narrow domains, with limited repertoires and a tendency to get stuck or do something odd. Coding help is the exception, and he called Claude Sonnet 4.5, released the day before taping, the best coding agent by current benchmarks. He expects the reliable version to arrive within a few years and thinks we'll all be using it.

Finance has had algorithmic trading for a long time, so it's the closest thing we have to a preview. Summerfield points to a flash crash where about a trillion dollars came off the New York Stock Exchange in roughly half an hour. None of the tools involved were especially powerful on their own. The dynamic effects between them ricocheted through the ecosystem and caused the damage.

Summerfield answered this one normatively, meaning he said what he wants rather than what he predicts. He wants clinicians and attorneys in the same roles they hold now, using AI where there is a clear right answer, and owning the value judgments. Nick put it in personal terms: with a possible cancer diagnosis he wants the doctor using AI for accuracy and a human being there to talk through what it means.

Summerfield says you can answer that in two flatly contradictory ways and both are true. Structurally they rhyme: neurons joined by synapses that get adjusted during learning, units joined by weights that get refined the same way. Underneath they're nothing alike, since one is fat and water and protein and the other is silicon, and the brain has many specialized parts while a network is largely undifferentiated. His line was that ChatGPT is very clever, but you'll probably beat it at tennis.

By some estimates, Summerfield says, around 30 to 40 percent of jobs are teleworkable, meaning you could do them entirely by talking to a computer. That leaves the rest, and he is blunt about it. Someone still has to pick the strawberries, service the power station, and weld things back together. He joked that hairdressing is where he'd go, since he isn't expecting a robot barber anytime soon.

Summerfield's argument runs through trust. Agency in a social setting has another name, and that name is trust, built between people who are fundamentally equal even when their status or wealth isn't. A relationship with an AI is not symmetric, because it's a consumer service from a large multinational corporation that doesn't know who you are. That's why he wants society's important decisions grounded in trust between humans.

Nick brought up Nassim Taleb's work on fragile versus robust systems, and Summerfield agreed that centralization begets fragility. He added that risk also rises when systems are densely interconnected and tightly coupled, with no slack. Today's AI is not tightly coupled yet, because there's no infrastructure letting agents intercommunicate and share dense information. He expects instability once that infrastructure exists.