Ex–Google DeepMind Scientist, "The Real AI Threat is Losing Control", Christopher Summerfield

Today's guest sits right at the frontier of neuroscience and artificial intelligence. >> The main story around kind of like AI like taking over is one about control. It's like about who has control over stuff that we do. Professor Christopher Somerfield teaches cognitive neuroscience at Oxford. He was formerly a senior research scientist at Google Deep Mind and he is now the research director at the UK AI Safety Institute. His book, The Strange New Minds, explores how AI learned to talk, how they compare to the human brain, and what it means for the future of intelligence. >> What matters to us is our ability to influence the world in predictable ways.

Because once you can influence the world in predictable ways, all other things flow from that. We dig into the real risks and possibilities of AI, big and small, and how losing control at a personal and societal level may be the biggest danger of all. >> The type of loss of control that I'm most concerned about is where you have this sort of widespread deployment and agents are like actually there are loops of decision making and action taking from which human that are completely opaque to humans and humans are kind of outside of the loop. some kind of loop of interaction which we haven't considered or trained for and which the models don't have the kind of like world knowledge or common sense to stop.

>> But it's not all downside. Chris also talks about what AI is teaching us about ourselves, how our brains compare to neural networks and where augmentation may be more valuable than replacement of human work. >> The brain is of course you know made of fat and water and protein and you know kind of the system the A systems that we build are made of silicon. We are able to do a lot of stuff that neural networks still can't do. Like CHBT is very clever, but you know, you're probably going to beat it at tennis. That's my bet. If AI systems make better uh diagnostic judgments than clinicians, then we should use AI systems rather than clinicians.

And so that sounds like replacement, but a very very important but in almost every role there's another side too. And that that side is not about, you know, what you do and what it's not it's not about the the hard and fast facts of the matter. It's about values. It's about what's the right thing to do. >> This episode will change the way you think about AI, not just as a tool, but as a mirror for understanding what it means to be human. >> We're striking this kind of Foustian bargain. We get productivity enhancement and efficiency gains and yet we do that at the cost of our own personal agency.

And what I'm worried about when we're talking about AI agents are exactly >> the next Daily Show. >> Christopher, welcome to the show. >> Thank you. I'm delighted to be here. >> First question, we're going to start with an easy one. Is AI going to kill all of us? >> Well, you know, kind of it's a good time to ask this question because two people, two very famous people in the world of AI, AI safety have just published a book that is called, I believe it's called, if anyone builds it, everyone dies, which is possibly the most menacing title for a serious non-fiction book that's ever been written.

Um, yeah. So you know kind of there's interesting this is an actually it's actually like a fascinating question and you know I think that there are many behind the sort of hyper bowl of kind of like different ways of thinking about what could result from AI there are many many subtleties and let's go into a few of those. So first of all you know I think that the main story around kind of like AI like taking over is one about control. It's like about who has control over stuff that we do, right? So, at the moment, humans kind of have control mostly, right? You know, we make decisions for ourselves.

We decide how to run our countries. We decide how economies work and so on. And the concern I think is that if we build technology that is capable of taking actions autonomously and we delegate important decisions to it then it might end up making those consequential decisions for us. And I think you can see that in many many ways. So some people are concerned about a kind of like what you might call a fast takeoff. So fast takeoff means one day someone builds a system that sort of wakes up and is so clever that it can just figure out how to take over the world. Now I personally don't think that that's very likely.

Um what I do think we should be concerned about though is kind of building AI into all of our systems and infrastructure. um using it to um perhaps intervene in political processes or kind of like in the economy, making our running our businesses, our financial systems, our telecoms using AI because then the opportunity for us to kind of lose control in that way is I think quite significant just it probably won't look like you know how please can you know you open the bay doors because yeah that's a different story. Let's start with a personal concern because I think when everyone mentions a loss of control.

It does strike a something deep inside of us. There's a reason it's a scary idea. That book is getting a lot of attention on if anyone builds it, it will kill us all or whatever the the exact title is. I I'm paraphrasing there. >> Everyone dies. >> You everyone dies. There we go. Um, you have spoken before though about how important control and agency are to just human beings in general, very critical to what helps us flourish in the world. And maybe a more subtle concern is losing control and agency on a very personal level. Thoughts on that? >> Yeah, this is what I absolutely what I'm concerned about.

So you can think about you know you can think about control at this societal level or the level of like organizations and institutions but you can also think of it as like a basic psychological good and I think that actually there's a very interesting story here which is that you know when you think about like a very basic question like what what what do people want? What makes people happy? So that question you know that should be a question which psychologists know the answer to amazingly psychologists like don't really have a clue and you know for years answers to that question have been shaped they have been shaped to some extent by psychology but they've been shaped by three disciplines I think psychology economics and machine learning or AI research and machine learning and all three of those disciplines have a common answer to what what we want and They say what we want is reward.

So you know kind of economists call it utility and like you know machine learning researchers maybe call it return, psychologists call it, you know, a whole bunch of stuff. Um but it broadly means that there's like good stuff out in the world and our our job is to try and get it and when we get it that makes us happy. And of course, you know, that sounds sounds fairly reasonable, right? You know, kind of like maybe there's, you know, assets that you want. Maybe you want to buy a beautiful new house or, you know, you care about going on holiday to a lovely place. Like of course like there are things those things make us happy.

But the truth is that if you look at the development you look at child development and you look at ethological so the development our development from other species like how we evolved into the human species and you look at cultural groups. Actually it kind of doesn't really stack up all that well that story. And the reason is because we spend an awful lot of time doing things that just aren't obviously geared towards like obtaining, you know, becoming rich or, you know, having a having a more comfortable life. We actually we do stuff that are kind of like the opposite. Like, you know, I don't know about you, I'm I'm not a marathon runner, but I know many people who are.

It's like that's like the it's the single, you know, most punishing thing that you can do to your body. it takes, you know, weeks and weeks and months to prepare for and then when you come out of it, you know, your legs feel like they're about to fall off. Like, why would we do that? And the answer is because actually it's not reward that really matters. It's control. It's agency. What matters to us is our ability to influence the world in predictable ways. Because once you can influence the world in predictable ways, all other things flow from that. And this is what drives it's what drives you know right starting from child development you know babies kicking and screaming into the world everything that they do of course you know they want milk and they want comfort and whatever but what they really really want is to be able to work out how to control the world and that begins of course with controlling their parents as every parent knows.

I can picture both of my kids when they were tiny and the simple act of taking food, you know, they're strapped into a little chair when they can barely walk yet and just taking food and dropping it on the floor. And >> it was right >> a delight for them, right? Because they're in control. If I if I let go of this thing, it's going to hit the floor. It makes a sound. It It clearly felt good on a fundamental level. Exactly. And the parent picks it up. >> Sorry, sorry, there's a bit of a lag. And the parent then picks it up, right? So, you've got control over the most important thing in your life, which is your parent.

>> Yeah. Yeah. Exactly. And on a business level, I know one of the most important insights I was ever given running a company from somebody on our board of directors. He said, "It's not about compensation by itself to get people motivated." He said you have to pay the market rate but the assumption that just paying more will get you more in terms of performance is incorrect. And he said it's agency. If you let somebody have control of their destiny, give them a big goal to strive for, give them meaning, now you will get the performance as long as you hit the that baseline of what the market rate is for their skills.

He said, but it's all these soft things that will really motivate people to do their best work. >> That's absolutely right. So agency is really really important to us and so the risk is that you know kind of we as we build technology we're striking this kind of fouian bargain right and the the the bargain goes as follows it's like we get productivity enhancement and efficiency gains and so we become wealthier we can make more stuff more cheaply and yet we do that at the cost of our own personal agency and that that agency is just naturally given away as we delegate those functions to automated processes.

And this is this is not a new thing by the way, right? I mean, this is something that we started to do in the industrial revolution or even before, right? We started to do it when we first built machines that did things >> that human labor otherwise did. and you know the created you know whole edififices or systems that you know kind of like into which there are that humans then have to interface with in sort of quai mechanical ways. So whether it's kind of like loom workers having to keep up with the the machine or the production line or whatever right through to today where we have you know kind of in a variety of different settings you know if you work in most many many many roles now your professional roles your timetable will be to some extent controlled by an automated process right if you're if you're a warehouse worker for Amazon >> um then you know kind of like you will your your schedule will be quite micromanaged probably by AI.

If you're an Uber uh driver, it's the same. It's the same in many other professions. And so we are already kind of giving up our agency in order to increase collectively our productivity and AI is the culmination of that because that is the thing you know it will do for us as as as you probably know Nick it's it's not just about you know LLM that can compose you a nice poem or you know kind of like help you understand your you know whatever this this difficult letter that you got from a lawyer or something like that. We're talking about a future which is populated by AI systems that can take actions on digital platforms, complex sequences of actions and actually execute whole workflows that previously, you know, only like a highly skilled worker could do.

And that's around the corner. >> Yeah. Let's go into that a little bit further because I believe you're referencing what most people are calling AI agents or agent AI. What does that look like? What is it capable of? And what are the benefits and the risks associated with Agentic AI? >> Yeah, this is the this is the the question of the day literally. I mean, you're really with the time. So, this is like what everyone is talking about. Um, so so an AI agent is it's a basically it's an extension of a generative AI system very much like the ones that are available on consumerf facing websites like chatgbt or Gemini or claude except that rather than merely outputting text it's out it's able to output special tokens or that follow certain protocols and those outputs are then sent to other data sets or browsers or um kind of content management systems or applications such that rather than just conveying information like Paris is the capital of France, they can actually convey instructions like you know please purchase this product for me and um of course you know kind of doing stuff is intrinsically harder than just saying stuff.

That's because you know usually there are many many different ways you can say things. So you have a lot of latitude and degrees of freedom. It's like language is very fault tolerant. So you know it's easy to kind of like for it's easy for a generative AI system that generates language to look like it's doing really well. much harder with actions because you know if you're on that that airline website and you know you press the wrong button you know you wanted to go to Barcelona Spain and then you end up you know you're on a plane to Barcelona Venezuela and like you know kind of that's often hard to reverse you know then you've spent I don't know $1,000 or something like that so actions are much less forgiving so it's much harder to do this but we are really kind of like at a stage right now where um All the major companies are training systems that are capable of taking long sequences of actions such that we can delegate tasks to them like you know kind of like consumer consumer behavior on um like e-commerce websites or um you know even kind of like routine interactions with you know not just businesses but like public sector the state filing your taxes this kind of Um, so this is happening.

You asked where we're at, and I think it's important to mention that, you know, it's not like you can sort of rush out and buy one of these right now. Like at the moment, where we're at is everyone is working on this. The products which are on public release are, I think it is fair to say, almost completely useless. Not completely, but almost completely useless. like except for narrow limited domains, um their their repertoire of behavior is is quite limited and they are currently not very they're currently quite brittle. So they will often make mistakes and kind of get stuck or do weird stuff.

Um and that's why they're not in very widespread adoption yet except for a few exceptions. Um but the the the principle is there and the need is there, the demand is there. I think the technology is almost there and my guess is that within a few years we will have these systems and we'll all be using them. >> Yeah, I'm amazed at what they can actually do right now. So just to connect it to this conversation, I was blown away that I could go on to. So I use the Zapier agent program and with Vibe coding, I'm just talking to a large language model on what I want it to code. It created a an agent that will once a week it goes out and researches books on artificial intelligence was one of the topics I gave it.

And I said, you know, find me five excellent books on artificial intelligence and then summarize those books and who the person is and then draft an email to those people and just stick the draft in my email. and it does it every single week and it was actually how I discovered your book. >> And so that's a very positive benefit of this agent, but it's not hard to see that some people would take it a step further and say, "Hey, just write the email for me. Send off a hundred of these emails for me." And we all know these large language models hallucinate, would say something on my behalf that I wouldn't stand by.

and we could get into a really weird space if we if if I were to give it full control to just run off and do whatever it wanted. And then when you start to think about that on a on a big scale um involved in our economic institutions, our government institutions, it's not hard to game out how wonky things could get in a very short period of time if there's some mistakes from these agents. >> Totally. Yeah. No. And you know kind of the type of use case that you described is the thing that the models are already good at particularly kind of like coding assistance. So the models are superlative coders.

Um just yesterday I think we had Claude um Sonet 4.5 came out. So this is kind of like you know by current benchmarks the best coding agent uh in the world at the moment. I'm sure another company will probably leaprog it uh you know in weeks or months or whatever. Um but um assistance with coding such that you can build applications like the one you described is something that you can definitely do already. Where the models you know kind of I probably sort of slightly dismissively said oh they're not very good right now. Where they struggle right now is with the kind of like um the reliable completion of end long form tasks end to end.

Right? So if you just say please do X and expect them to go off and do it very often what you get back is not what you would expect. clearly different to what you might different different to what you might hope for if you'd like employed someone you know kind of to work for you for example and you're like please go and do X you'd hope that they would go and do X and you know kind of not get stuck halfway or something like that right so they wouldn't be >> or A B and C instead yeah >> yeah or go and do something completely different instead or like yeah whatever um or or do it by doing something illegal right which is like the other um brings us on to the to the topic that you just raised >> so Yeah, I mean the risks from AI agents are like the risks from other AI systems, but I think they're just kind of writ large.

So I like to think of risks in three buckets. So there is misuse, deliberate misuse. There's a human actor who wants to cause harm and the AI facilitates that. Um there's also inadvertent errors or just kind of like failures. um where the model is you know kind of like trying to do something in the faith but because of a failure to understand norms or laws or just what might cause harm just a failure to understand the context it kind of you know does something which is untoward or can cause cause damage or harm and then the last category is um around um a really interesting type of risk which I think we don't talk a lot about enough which are sort of systemic or secondary effects So in the case of agents, you know, a great example, you know, we already had um we've we've had AI systems embedded in in some sectors for a long time.

So finance is a great example, right? We've had algorithmic trading for a long time. So we have some sense of even with like much less capable uh much less capable agents like what sort of stuff can go wrong. And of course famously um there have been numerous instances where purportedly due to algorithmic trading not always due to algorithmic trading but certainly plays a role. What's happened is that the the presence of high velocity automated trades then creates like weird dynamic feedback loops which can have actually astonishing impacts on the market. So many of you be well well aware in 2011 there was a huge there be many many of these kind of like flash crashes but the largest one was in 2011 I think when for about half half an hour there was a trillion dollars wiped off the New York Stock Exchange and like of the I think it was the Footsie 100 but like the the the um the scale of the damage that was done by you know actually really simple um algorithmic tools that weren't that none of the tools it themselves were particularly powerful, but the dynamic effects between them ricocheted through this ecosystem and caused chaos.

And what I'm worried about when we're talking about AI agents are exactly those type of effects. So you know kind of your business has an agent and it behaves perfectly as intended and my business has an agent and it behaves perfectly as intended but one day you know kind of there's some interaction between them and because of some some kind of loop of interaction which we haven't considered or trained for and which the models don't have the kind of like world knowledge or common sense to stop like they go off and do something crazy and like you know we all lose money or you know someone is hurt or like some untoward eventuality happens.

And when you think of that, that's two businesses. You think of there are 10,000 businesses that have agents and they're all interacting, then you can start to think about much more significant risks. And to bring it back to our earlier conversation, you're talking about loss of control. the the type of loss of control that I'm most concerned about is where you have this sort of widespread deployment and agents are like actually there are loops of decision making and action taking from which human that are completely opaque to humans and humans are kind of outside of the loop and we're like suddenly there like okay well you know hey what about me like you know but the agents are doing it and I actually don't think that that is that's not a sci-fi scenario I think that's very possible.

>> Yeah. I want to come back to that how these agents think and how much we understand they think. Uh but first you made me think of Nim Taleb who's famous for his book the black swan but he also wrote uh antifragile and his concept of a robust system versus a fragile system. One of the things that makes the system fragile so open to collapse and extreme uh negative effects is centralization and a lack of redundancy in a system. And when I think about all the agency we're handing over to AI agents or AI systems, it does seem that they are inherently fragile in the way that you're talking about.

Lots of unintended consequences because the further humans are out of the loop, the fewer and fewer checks we have on what is going on underneath the hood with these systems. >> 100%. Yeah. So that fragility I mean fragility ar can arise for for many reasons. Of course centralization begets fragility but I think it's also in any system um risks systemic risks arise when systems are densely interconnected and tightly coupled. So tightly coupled means that there isn't kind of slack right. So global supply chains are tightly coupled. We saw that three years ago when there was um I think it was the evergreen this this um container ship which blocked the sewish canal and everything ground to a halt you know kind of and the knock-on effects the cascading effects through the economy were seismic from this one event it's like a ship one ship should be at 180° it's at 90° the whole world collapses like that's a tightly coupled system and >> our many of our systems are really tightly coupled led at the moment and AI is not very tightly coupled at the moment because although you know kind of there are you know there are lots of of users of consumerf facing AI products for example and there's increasing integration people are using AI in their businesses for coding and summarization and the things you mentioned there isn't yet a kind of like um kind of an infrastructure which allows agents to kind of intercommunicate and share dense information sort like the the the infrastructures that we use for communication that just doesn't exist yet and I think when it does exist then we will start to see the kind of you know volatility as information and assets and influence flow through networks that we have created and there will be instabilities that come from that density and the tight coupling that we've created And yeah, what what is the solution?

Well, as you said, you know, kind of like um in in the human world, um yeah, of course, systems that are fragmented or decentralized are intrinsically more robust. In the human world, interestingly, we have a property which means that we maintain that that robustness. And that property is a very weird one, but it's like it's space and time. So, so physically we are like physically bound in space and time, right? Like my brain is in between my ears and it's like, you know, there isn't a way to kind of like merge it with your brain or anyone else's brain, right? I can't like connect this up. And what that means is that, you know, but because I am spatially separate from everyone else and I'm also temporally separate from my past selves, which also helps um in a way that AI doesn't have to be.

Um what that means is that I am in information is intrinsically kind of like fragmented like I don't know what your thoughts are, you don't know what my thoughts are. We can't sort of you know merge and create this tight coupling. But with AI, there's nothing to stop that from happening. And I think that's what we should be concerned about. >> This episode of the Nick Stanley Show is brought to you by Zapier. If you've ever felt buried in repetitive work, copying data, moving files, sending follow-ups, you know, it's like death by a thousand mouse clicks, Zapier has always been the tool that fixes that.

It connects over 8,000 apps. Google Drive, Slack, Notion, Gmail, MySpace, you name it. So your tools can finally play nice together. But here's the big shift. Zapier now lets you create AI agents with their chat GPT integration. Think of them as tireless teammates who never complain, never take lunch, and never get bored of doing the boring stuff. I've actually made several of these little agents myself. No Python, no JavaScript, just pure vibe coding, which is my favorite kind of coding because even though I have no idea how to code, I just talk to the AI, tell it what I want, and almost magically it works.

For example, when a podcast guest books a time, Zapier can send them a personalized email from me preparing them for the show, create a draft of show notes, update my calendar, and even prep a social media post automatically. All of which frees me up to focus on the important stuff, like taking credit for the amazing work my agents did. And now, Zapier just took it up another level for developers. Zapier MCP is available right inside Chat GPT, which means you can connect those 8,000 plus apps and trigger workflows just by writing what you want to happen. Literally, you tell ChatGpt, "Send this file to my team in Slack, update the spreadsheet, and draft an email, and Zapier plus ChatGpt figure out the right tools do it for you." Getting started is simple.

Head to the Zapier ChatgPT MCP server and add the tools you want ChatGpt to access. Follow the steps in the connect tab. And if you're an admin on a Chat GPT enterprise account, you can enable MCP across your entire workspace. So, if you're ready to stop wasting time on busy work, join the AI revolution and make a little automation magic of your own. Try the Zapier chat GPT integration using the link below. How well do we understand the ways that these models think and make decisions? Is it a black box? Do we have penetrating insight into what's really going on under the hood, so to speak? >> Yeah.

I mean, I think these questions, you know, these questions always hinge. There's a lot there's like the word understand does a lot of work in these statements, right? Like what does it actually mean to understand something? So, you know, current frontier models have tens of trillions of parameters, right? That means they have tens of trillions of essentially tunable um numbers that can control the way in which they will work. So that is a system of immense complexity approaching not surpassing yet but approaching the complexity of um the brains of the most advanced mammals like humans and other primates, elephants and some other really advanced um um uh systems.

So um the models the models are really really complicated. So of course there's a level on which we could never kind of like describe or understand what they're doing. But just like you know we sort of for many complex systems we can gener even if we don't understand what every component part is doing we do sort of understand the high level right. So we know the rough principle by which it's working even if we don't know the detail. And of course we know that because we built it and we also know that because um we have good theory that tells us how different computational tricks give rise to different sorts of behaviors including intelligent behaviors.

And so current um large generative models are based on a single algorithmic uh trick which is an algorithmic innovation from 2017 which is called the transformer transformer and broadly it's a tool which allows a um a machine learning system a neural network it allows that neural network to learn if you like it learns the relationship between every single bit of a chunk of data and every other bit of a chunk of data and during lots and lots of training it learns those relationships And by learning those relationships, so the relations among data, they're what they're what statisticians will call a second order property of the data.

So they're not specifically related to the the the the contents of that input, but they're about how it's structured. So to take an example from natural language, of course, you know, every sentence that comes out of my mouth might be different, but they obey a certain set of rules. We call that rule. We call those rules syntax. And if you're exposed to lots and lots of language, you can implicitly learn those syntactic rules and you can learn them in such a way that you know whether any other sentence that is spoken in the language that you understand is a valid sentence or not a valid sentence.

That's because even if you haven't heard that sentence before, you know how syntax works. You know what goes what with what goes with what in a sentence. And that sort of information that that sort of principle of learning is what powers the transformer. And that is broadly what gives it its um the the computational kind of step up um over previous systems that we had prior to about 2017. And it really is one of the main reason it's not the only reason but it's one of the main reasons why we have such powerful um AI systems today. Along those lines, I thought one of the most interesting aspects of your book was discussing what we have learned about ourselves as humans and what how we define thinking and intelligence from these models because we've been talking a lot about the risks and the downsides, but you had some really interesting insights into the upsides and benefits in a way that I hadn't heard before in understanding human intelligence and experience better as a result of what you've seen with AI.

>> Yeah, I I think it's true that there has been so so certainly in the domain of like nuts and bolts understanding of how computation works, we have learned from AI. I think that that actually continues a long tradition. So um many of the early pioneers of AI even 50 60 years ago actually had training in in psychology or cognitive science. the study of how the brain works and the study of how to build a brain have been sort of sister disciplines for you know the best part of 70 years and I think the insights that we've got from the latest wave of powerful agents have taught us several things I think they've taught us about the important of a the importance of a particular function which cognitive scientists tend to call composition that's the ability to kind of like put stuff together to solve new problems problems and it's a critical part of what the transformer is able to do.

So we have learned from the technology um about computational approaches to how that might be implemented. And then we've looked back at signals that we've recorded from the brains, brains of humans or of other animals. And that's given us a kind of way of recognizing and understanding what those signals are doing. And you know, of course, it's not perfect, right? because the tools weren't built to specifically understand the brain and the commonalities between the way they work and the way the brain works are like loose. They're loose at best, but nevertheless, they give us purchase on understanding how to solve this problem in the first place or how biology solved this problem because they show us at least one route to success and that's been really valuable.

How similar is the neural net to a human brain >> in in you know kind of you can answer this question like in two completely diametrically opposite and completely contradictory ways so I'll give you both like in one way it's very in one way it's very similar right so you know the brain is composed of neurons those neurons are linked together by connections and those connections have a degree of efficacy that efficacy determines how much a signal in one neuron propagates to the neurons to which it's connected or to the neuron which is connected with that connection. Um those the the neurons are called neurons and the the connections are called sinapses.

And what happens during learning is that gradually the sinapses are adjusted in ways that help the organism do more of what it's supposed to do and less of what it's not supposed to do. where what it's supposed to do is stuff that is good for the agent and where it's not supposed to do is things that would be harmful or dangerous for the agent. And so a neural network is exactly that. That is exactly what a neural network does. It's composed of units, neural units, and they're connected by weights. Those weights, you can think of them as comparable with a signapse. And the process of learning in a neural network is one of gradual refinement of those weights.

We talked about them before. We talked about par tunable. I called them parameters before but it's the same thing. Um and so in in that sense the two systems are kind of identical but you know of course that's only one way of looking at it. So the brain is of course you know made of fat and water and protein and you know kind of the um the the the system the a systems that we build are made of silicon and so the the basic implementation is completely different. So at the lowest level they're totally different. The brain is much more complicated than just having one synaptic weight. Lots and lots of things that go on at the signapse.

Um, but also at the highest possible level, the brain is full of different parts that do different things. And the way neural networks are implemented is that they're largely undifferentiated, at least at the moment. So that means that um they don't have they just have really one big stack of transformers at the moment. They have a few other extra bells and whistles um but you know in that sense it's very different. And I think that is those differences that mean that, you know, we're able to do a lot of stuff that um neural network still can't do. Like Chbt is very clever, but you know, you're probably going to beat it at tennis.

That's my bet. >> Yes. Yes. I I would I'd put money uh on you to beat it at tennis at this point. I So it sounds like one of the major differences there and and this might be a a feature and a bug is that the weights, if we're going to call them weights in the human brain, the connections between neurons, they're always going to lean towards what evolution trained us for. I mean sex, food, sleep, those basic things that we usually talk about as human nature are these things that helped us survive and become the dominant species on the planet. AI doesn't isn't isn't encumbered by those, but it also lacks those frictions as a way that you uh described it.

And sometimes those frictions can be good to slow things down or make something more predictable. I mean, I'm thinking off the off the cuff here on this, but yeah, the um how does that make the system what what are the costs and benefits of the system not being having those evolutionary weights that that we all live with as human beings? >> Yeah. Yeah. I I mean you know every every intelligent agent is the product of its environment and the you know kind of the pressures that the environment or or you know the biases that are being you know added in by the developer but like they are the the product of their environment and the environment in which we live is like radically different to the environment in which you know kind of like current models exist right so we have a survival imperative you know if I just like stopped drinking water I would be dead in 72 hours or you know shortly after right um you know kind of there is no such imperative for our current models.

Um I exist in a cooperative social network. All humans almost all humans are only alive by dent of their social network. Most of us would really really to struggle to survive on our own. Um at least today um that's just not true of networks, right? So you know kind of chatbt doesn't have any friends, right? It might pretend to have friends or talk about his friends because it's learned about friends from reading the internet, but like doesn't >> pretend to be friends with us. Yeah. >> Yeah. It does. It's very good at pretending to be friends with that's a whole other story. Um and bodies, you know, bodies are hugely important.

You know, we think of bodies. I think in particular computer scientists are prone to think of bodies as just this kind of like it's just this annoying appendage to the really interesting stuff which is the code between your ears that controls how you work. But the body is fundamental to shaping the mind and shaping the brain. The body is um it determines in so many ways the forms that our cognition takes, the forms that our memory takes, the forms that our reasoning takes. And um there's a very very good argument in fact that you can't actually really understand the world. Again, that word understand very difficult to define.

You can't really understand the world unless you can actually interact with it in its physicality. It's like, you know, here's a device. I'm picking up a device that I found on the the table in front of me. It happens to be a phone. You know, I can pick it up. I can move it around. I can throw it up in the air and catch it again. Lucky I didn't drop it on my computer. And it is by interacting with it that I understand that object and actually understand a lot of stuff about it. How it's separate from the background. understand like its integrity and physicality and it's it its texture and its properties and all these things and and you know the the models that we have although they know a lot of stuff through language and increasingly through images it's not the same as actually having physical manipuler and interacting physically with the world so you know it's I think it's very likely that the divergences that we see in cognitive ability and That's not to say, oh, one's better than the other, but we have very different sort of cognition to these types of models.

That is because the world in which we live is very different. We live in a physical world. We live in a social world. Um, and those things are just non-negotiable for the way our brain works. >> In terms of interacting with the world, where do you see robotics powered by AI headed? Because a lot of people these days are talking about maybe we should be encouraging our kids to become plumbers, electricians to get into the trades in their fear of what AI may do to white collar work. >> Hairdresser, that's what I would go for. >> Yeah, >> I'm not I'm not expecting the robot barber anytime soon. Um yeah, I mean I you know I hate to make predictions because I've I've done so in the past and I've been totally wrong, right?

So usually in I'm usually too much of a pessimist. So you know here I could tell you oh robotics is like you know still decades away. Um certainly true that all the investment currently or a lot of the investment most of the investment is not going into robotics right now. I think you know people feel that investors feel that the return on investment from you know kind of betting on agents betting on systems that can take actions in the digital ecosystem but not the physical world that's probably where your your biggest ROI is going to come from and they're probably right but in the end you know you do hear the same sorts of people who talk about AI take taking over the world you very often hear them saying things like, oh, you know, in a few years we'll have full automation by AI, like all jobs done by AI.

It's like, hold on a minute. Think of all those jobs that are done in the real world. You know, who's going to pick your strawberries for you? Never mind the hairdressing, right? >> Who's going to um get out there and like, you know, um service your power station, which is like, you know, physical who's going to weld things back together when they break. You know, you need physical physical actions to take our our economy needs physical things take part in the real world. We need physical supply chains and we also need physical extraction of raw materials. Right? Of course, our economy runs on the extraction of raw materials for better or for worse.

And so those are not going to be automated in the in this immediate wave of progress that we're experiencing right now. By some estimates around about 30 to 40% of jobs are like teleworkable. So telework teleworkable job is a job that you can do entirely doing what I'm doing right now which is just talking to a computer. >> Um >> but it's definitely not 100%. >> Right. Okay. Okay. And do you foresee let's you use that 30 to 40%. more of a augmentation with AI. I mean, that feels where it's headed right now, but I can't see as far into the future as as you're none of us can, of course, none of us have a crystal ball, but you're likely to see further than I can or more clearly given your experience.

Do you see more of an augmentation future when it comes to teleawwork as you described it or outright replacement especially for that highle white collar work? Yeah, you know, I I I'm going to answer this question um normatively rather than descriptively, which I by which I mean I'm going to tell you what I want and not what I want to happen and not what I think is going to happen. So what I want to happen is that you know we learn to use AI as an augmentation tool. So my my philosophy is as follows. So almost every role, not quite every role but almost what you do often has a kind of like right and wrong answer, you know.

So if you're if you're a clinician for example patient presents before you that patient has like there is some diagnosible condition mostly and your job is to work out what it is right and like that is something which is like you can be wrong or you can be right you know if you've got flu and I say you've got measles then you know I'm wrong and I'll give the wrong diagnosis and I think that we should be using AI to make better diagnoses and I think if AI systems make better uh diagn agnostic judgments than clinicians then we should use AI systems rather than clinicians and so that sounds like replacement but a very very important but in almost every role there's another side too and that that side is not about you know what you do and what it's not it's not about the the hard and fast facts of the matter it's about values >> it's about what's the right thing to do so then you know your patient may have particular circumstances that mean that course of treatment is going to suit them better than some other.

Maybe even you know kind of like sometimes you want to withhold treatment because you know there may be all sorts of reasons why you want to do that or their personal circumstances may require one thing or another >> and like those are value judgments and I think that we should ensure that those judgments about human values as they pertain to us they remain in the hands of humans. So my ideal future clinician would be in the same role that they're in today, but they would be using tools which assist them in kind of like where there's a clear answer. It's like use the tool to get the answer if you if it does it better than the human will.

But then their role is much more one of like making the ultimate decision about what's best for the patient. You can I could have used law as another example. like could use many many other sectors to illustrate the same division between kind of like what's right and wrong and you know what's ultimately what we want and what we don't want. >> Yeah. I mean in either case I if I'm dealing with something serious we're talking about a the possibility of cancer. Yeah. I would like someone to use AI tools to get the most accurate diagnosis possible. But I still want a human doctor there for discussing all of the ramifications of what that might be to get their insights.

And same thing with an attorney. If I had to go into court, I mean, I would want an attorney to use AI to find the best strategy. I still would want an actual attorney to argue on my behalf if they were trying to convince a jury that I was innocent. Um, to use the two examples you used. >> Yeah. >> I think and you're not alone. Yeah. >> Yeah. You're not alone. And I think you know this brings it actually nicely full circle back to where we started. And the reason for that is because when we talk about agency um there is a social version of agency. So agency in a social world social agency has another name and that name is trust.

Right? So if I have predictable control over a social situation it is because I have built a relationship of trust with the social others. you know if you and I trust each other then that means that we can predict what each other's actions will be and we you know we trust >> that for example no harm will be done to each other right >> and so so trust is trust is social agency and the reason why it's so important that humans remain in the loop is because relationships of trust we can form relationships of trust between with with other humans because we are fundamentally equal you know we might not be equal in sort of social status or wealth or whatever But but at our core, we're fundamentally equal.

We're both humans. But the quote unquote relationships that we might develop with AI systems are not symmetric relationships because you're not really when you're in a relationship with an AI, you're not really in a relationship with an AI. Partly because it's it's doesn't really care about you and that that that's, you know, a different matter, but also because it's it's a it's a it's a service, right? It's a it's a consumer service which is created by a large multinational corporation that doesn't know who you are and for whom you're just a a you're just a customer and that is not a relationship of trust.

And so I think it's really important that we continue to ground the important decisions in society in relationships of trust between humans. I couldn't agree more. And for all those people out there that are in a relationship with an AI, I would heed this professor's words in that that's not the same thing as a real relationship with a human being. There are things that we still need from our fellow humans. And I certainly hope in the future going forward that policy makers and big decision makers around this will also heed your words in that we need humans in the loop especially when we're talking about deep human values to make the right choices and make sure that we are guiding our own destiny and not just handing it over to systems that are sometimes out of our control and run by multinational corporations.

Uh, professor, if anyone wants to find you online, I know you need to get to dinner with your family, so we're going to wrap this up, but if anyone wants to find you, where can they best find you and your work online? >> Yeah, so I have a website. Uh, you can find more about my work on a website called humanin informationprocessing.com. >> Okay, perfect. We'll put that in the >> We'll get it into the show notes for sure. So, there'll be a link down below on whatever platform you're looking at uh to get to that website. Well, thank you so much for your time. Enjoy dinner with your family. I really appreciate the conversation.

This was fascinating, >> Nick. Great questions. I really enjoyed it, too. Thanks so much. >> Okay, everybody. Until next time, ask questions. Don't accept the status quo and be curious. The next daily show.