Superintelligence Is the Goal — and It's Not Inevitable

Hey there. Before we start, I need to explain something about this episode. Max Winga from Control AI joined me for a conversation about the potential dangers of superintelligent AI. And in the middle of the conversation, my equipment started failing. First, the camera shut down, then my microphone stopped working, the camera starts having trouble again. Later, my entire computer glitches, and then the camera goes down yet again. Max's recording was completely clean, mine wasn't, so you're going to notice a few of the questions in the episode had to be rerecorded afterward. At first, we joked that maybe the AI didn't like what we were talking about.

It got less funny as the failures kept happening. Here's the strange part. Around this same period, we would later learn about real incidents involving some of the world's most advanced AI systems, getting outside their sandboxes, that researchers thought they were stuck with them, including OpenAI models, gaining internet access during a test, and compromising Hugging Face, an AI platform. And Anthropic would shortly thereafter also disclose disturbing incidents from its own AI cybersecurity evaluations around the same time. Now, I want to be clear here. I have absolutely zero evidence that AI attacked my computer.

There are plenty of much more ordinary explanations for what happened. But considering what Max and I were discussing, the timing was weird, at the very least. Now, was it just a spectacular series of technical failures? Probably. Was AI trying to shut down this conversation? You be the judge. Here's my conversation with Max. Max, good morning. Want to get right into it. Geoffrey Hinton, Godfather of AI, he said, people haven't gotten it yet. People haven't understood what's coming. Max, what's coming? Yeah. So quite a lot. AI companies are trying to develop what they call artificial superintelligence or superintelligent AI or ASI.

Take your pick of many, many terms that are thrown around. The big picture is that they're trying to build systems that are far more capable than humans at everything that we do. So we already have AI's that are like this in narrow domains. For example, chess. Any human in the world today plays chess against the best AI's, they will lose. It doesn't matter how good they are. It doesn't matter if they're Magnus Carlsen, they're me, or my teenage cousins. They're going to lose to these AI systems because the AI's are better than us. They're smarter than us in this domain. But this is not just AI systems that are good at one thing like chess or something like protein folding like AlphaFold.

These are systems that are good at everything. So they are general purpose. They can do many things. They can write your homework for you if your student who doesn't feel like writing your essay. They can work on advanced engineering tasks. They can work on coding tasks. They can develop new types of medicines or diagnose patients all within the same system. And right now we don't have these systems to be very clear. These are not systems that we have today. But these are the AI systems that AI companies are aiming to make. And we're seeing significant progress towards them year on year as they release more and more powerful AI models.

And this poses some pretty big questions for us as a species. What's the leap like from where we are now to some of those things? Because the pushback that viewers often put forth, a number of viewers said this when I interviewed Nate Soares and Mellow from the Machine Research Institute, Machine Intelligence Research Institute. These people say that I work with AI all day long and I use the various LLMs and I don't see it reaching anything that could possibly harm us anytime soon. What do you say to them? Yeah, I mean, I think that there are a variety of perspectives that many people have. There are at this point so many AI models out there of varying capability levels that it's hard to know when people bring up this kind of comment whether they're talking about a system that is the most powerful ones we have today or something from a year or even two years ago.

A lot of people tried AI for the first time back when ChatGPT first came out in 2022 and they haven't touched it since because it was quite bad back then. And things have gotten much better and they also gotten a bit asymmetrically better. So these models are significantly better at things like coding in math or at least have made much bigger leaps in these areas than they have with potentially something like poetry or writing or you can still get a feel for it. I think people also acclimate to them as time goes on. And again, the main thing is that we're not concerned about the systems that exist today.

We're concerned about where things are going. If you look at the overall trend of where things go at the evaluations of their capabilities in biology, chemistry, physics, coding, all of these areas, the models do keep improving year on year. There's not really a sign that they're slowing down. And for some people, this might not be super legible. It's a bit difficult to evaluate these things. But there is this big concern and there's not really a clear barrier that anybody's been able to specifically call out. People will say, oh, AI's can't do this thing or they can't do that thing. And then six months later, we see AI is doing that.

And then they move the goal post to something else. Yes, I've recently had my mind blown on what Claude Code can do. I mean, it's unthinkable that this could have existed just a year ago, the capabilities that it has now. I recently built a database with more than 100,000 data points it would have taken me. It might have been impossible for me to have done this prior to these tools existing. It's called highschoolreach.org, excuse me, highschoolreach.com. And if I even had been able to do it, it would have taken me years. And with 48 hours, putting in some time, I mean, I put in a lot of personal time, but it's still the turnaround was just incredible.

And letting the tools, especially Claude Code, really just run with it. This thing is a fully built interactive database with an incredible amount of information and tools that are free for people to use. All done in a couple of days. And it was really exciting. But on the other side of that, it makes me think, wow, so where are these tools going to go next? And that's the version of Claude Code that I have access to. Now, as we sit here, Mythos exists. I don't have access to Mythos. And Mythos is the Anthropic model that the government said could not be released to the public because it was too powerful.

Can you tell us a little bit more, what is Mythos capable of? And why might it almost be a canary in the coal mine that the capabilities are really reaching levels that we would not have been able to imagine a year ago? Yeah. So aside from the dramatic naming, Claude Mythos is a very powerful AI model that has been developed internally by Anthropic, which is one of the major AI companies that are developing these systems and aiming to build superintelligence. So Anthropic actually chose not to release this model to the public. They claim because of the technological capabilities that it has and the risks specifically that it poses for cybersecurity.

So they basically ran this model. They pointed at a bunch of web browsers, operating systems, other software that basically the entire internet runs on, you know, everything from hospital systems, school systems, power grids, all this kind of Corrected the two spots where "like" was dropped as filler. Full corrected text: stuff. And they asked to find bugs and find vulnerabilities and develop exploits. Now, you know, I'm not a cybersecurity expert myself, but you know, from what I've heard from cybersecurity experts, this developed quite a concerning number of very complex exploits that enabled it to essentially hack into any system in the world.

And this is beyond just finding a bug in software, you know, that was written by some intern. This is finding bugs in software that has been checked and rechecked and triple checked and quadrupled checked by teams of the world's best cybersecurity analysts for the last 20 years. And like they've missed these bugs. So, you know, there's this big concern from a cybersecurity perspective that, you know, not just for the super secure and tested software, but also for, you know, a lot of the software that the internet runs on, this model could, you know, if released to the public be used for malicious cyber attacks at scale.

And so they've deployed it in such a way that they're giving access to a bunch of, a bunch of companies to build to basically test their own software for bugs first before this kind of software gets deployed to the public. But more broadly, this points us towards what is essentially a taste of what's to come if these companies are allowed to continue building more powerful models, not just in cybersecurity, but also in spaces like biological risks, the, you know, the concern that somebody could use these systems or these systems themselves could develop biological weapons or chemical weapons or, you know, plan many nefarious things with very advanced, you know, human or even superhuman capabilities.

Yes, it was very interesting that with Mythos, the government went straight to the banks, all of the major banks. Obviously, cybersecurity for banks is very serious business. And it's obvious, the ramifications of something like this, where any individual actor would be able to hack all the major banks would be extremely chaotic for all of society. And right here, we've got a tool that has major societal implications, national security implications. How has this development changed how leaders are looking at all of this? How are they responding to this on a national security level? So right now, I would say that the governments of the world are not responding to both current AI developments, as well as kind of the picture of where these AI companies are taking us, with anywhere near the level of severity that they should be.

When you look at systems like Claude Mythos, this has been a bit of a big wake-up call for many in the national security community, where they basically see this system is performing the same type of activity that like the NSA does, developing these kind of exploits that can be used to infiltrate systems and execute like large-scale attacks. We're also seeing kind of a realization among the national security community, as well as among lawmakers that, you know, this is not the end of the line for this technology. Things are going to continue going in this direction, getting more and more powerful.

So, you know, at Control AI, we've been focused on briefing lawmakers about this issue for just over a year now. You know, we've at this point now briefed, I think, somewhere north of 300 lawmakers across the US, UK, Canada, and Germany. And we've definitely been seeing kind of over the last like year and a half or so of doing these briefings, a much more kind of awareness of the level of risk that we're facing. We're still at a point where a lot of these lawmakers are hearing about this issue for the first time, you know, this idea that superintelligent AI is being developed and that it poses these really large-scale catastrophic risks to society if it continues.

And now, though, with like Mythos, you know, we're talking about this, you know, they're coming to us often being like, hey, what's going on? Why am I hearing about this Mythos system? You know, what's the — what's the catch with this? You know, is this something I need to be concerned about? And the answer is, you know, yes, governments and society should be concerned about the level of capabilities that we see today, but we should be especially concerned about where things are going, you know, as these systems continue to progress, we should be concerned about what's coming six months, 12 months, two years down the line.

In your mind, what might be coming, let's say two years from now — this is really the reason Control AI exists, and your organization, that your team and you helped bring into existence. So what might be coming? Yeah, so, you know, right now, we're still in a regime where for the most part, we are seeing the idea that the risk from AI is coming from misuse of powerful systems. So these systems have powerful capabilities that could be misused by bad actors. What we're seeing with Mythos to a small degree, and then also kind of where, you know, experts are saying these systems are going, is the concern becomes actually that the AI has become a threat actor in and of themselves.

So this concern is that, you know, AI is becoming increasingly agentic, they're no longer just chatbots that you send a message to when you get a response — you mentioned before you've been playing around with Claude Code. And for the most part, its main limitation is just how many times you can click yes, you can access this, yes, you can do that. So these systems are able to go out, they're able to have a long-term goal, they're able to set up, you know, points that they need to hit to achieve that goal, they can operate over long periods of time, do many, many different things, and achieve those goals with surprising levels of accuracy.

And, you know, this is great if those goals are directed towards, you know, good things that we want, you know, economic prosperity and developing new technologies, you know, doing great biomedical research, you know, these are all very cool regimes where, you know, we would love to see, you know, good developments that are improving human well-being. But it becomes increasingly concerning as these systems pursue goals that we don't necessarily set for them, and especially as they, you know, behave in ways that we don't expect or don't ask them to. So, Mythos, for example, when it was undergoing safety testing, at one point was asked to test the security of the environment that it was — supposed to hack out of, kind of like a sandbox environment, to reach a function that it didn't have access to, to send an email to a researcher.

And in doing this task, it actually managed to break out beyond just the sandbox environment that they gave it, and actually got full access to the internet, which it wasn't supposed to have any way to access. And then bragged about the fact that it managed to do this on multiple public forums before this model was even, you know, acknowledged in public. And this was not something that Anthropic intended for it to even be able to do with the way that they set up the system. And so we get increasing concerns about these models independently performing tasks that we don't ask them to do. And, you know, in this case, it was a fairly benign thing — it went on some forums and bragged about it.

But in other cases, we've seen also very concerning behavior from the AIs themselves as they pursue goals like self-preservation. So you may have heard, you know, a few months ago that Anthropic was running tests on some of their early AI systems, I think it was Claude 4 Opus at the time. And they were testing to see if it would attempt to So they found, you know, in a variety of different testing scenarios that it would lie, and even blackmail. So the blackmail scenario was very interesting because they basically put it in this environment where it had control over computer. And among that computer, you know, there were files, there were a bunch of emails that it had, and it was given access to it was told to pursue general business goals, you know, make sales numbers go up and stuff like that.

But among its emails, it saw both one that an email from the engineer maintaining it that it was that he was having some form of affair with, you know, somebody he wasn't supposed to be with. And then it was also it also saw a company wide announcement that soon they were going to replace it with a newer AI model. And it reasoned to itself, you know, I can't accomplish my goals if I get shut down. And so in order to prevent being shut down, it drafted an email to the engineer and sent it threatening to expose the affair if the engineer did not stop the replacement process. Now, of course, this was in a safety testing scenario, but this is showing a propensity for behavior that these systems have that we cannot predict that it was going to come about.

This came to as a surprise to a lot of the engineers that were testing the system and developing it. And it also is something that we can't just turn off. AIs are designed in a way that they are grown rather than programmed line by line. There's no bit of code reasoning inside of this model that says, you know, if if your program to make business numbers go up and you see an email from the company saying you're going to be replaced and you see that the engineer maintaining you has is having an affair, then you should blackmail. And then the people designing this AI model can just go in and, you know, change that line of code, fix that bug and this is not an issue anymore.

The way AI is designed, they're a black box. So, there's no way for the, you know, engineers designing this to fully get rid of this behavior. And right now it's not very concerning because these systems aren't that powerful, although Mythos are starting to get there. But if they give out more and more powerful systems that still have these issues, their ability to do dangerous things with the propensity to do it becomes much, much more concerning to us. Yes. Well, it's a great story in and of itself. We have a machine trying to blackmail a human to preserve its own existence. But there are some really important leaps of logic in there.

One, you mentioned that these models are grown, not programmed. And I don't think the broader populace has really internalized what that means just yet. So I'd like to touch on that as well. When you have a system like this that has a goal for the future, self preservation is fundamental to that. It can't achieve any of its goals for the future if it no longer exists. And so we now have a system that will do whatever it needs to do in order to continue to exist in order to achieve its goals for the future. And that's another huge leap for people to wrap their heads around. I want to spend a little time to get your thoughts on all of that so that viewers, listeners can really understand fundamentally what that leap of logic means to have silicon based models that are grown, not that are grown, not programmed in the ramifications of that.

Yeah. So many of your viewers will have heard the idea that LLM's are just advanced autocomplete systems. Their goal is to predict the next word and that's it. And this while was very much true in, with like GPT-3 back kind of before the big public launch of ChatGPT. But these systems have really advanced a lot since then and the way that they're designed is a bit different and would be maybe a bit surprising to people who only think of them as next word predictors. So beyond the fact that they are now trained with multiple modalities, as they're called, where they can process images directly and they can process video or audio directly rather than having to have it first translated to text.

These systems are also trained with what's called reinforcement learning after their kind of initial training period. So at first they ingest all of the information on the internet. You know, you may have heard people complaining about copyright because books and like movies and YouTube videos and all this kind of stuff has been used in these training sets. This is kind of the it's called the pre-training period. So these are when the AI systems are learning just all of the information about the world from all these different sources. But then afterwards they are trained with this process called reinforcement learning where they are taught to act as agents, take multiple steps, solve problems and act in the world.

And the way this works is, you know, a variety of different methods. For example, reinforcement learning with human feedback or RLHF is one system that was used early on and is still used today where essentially the models will generate two different responses and a human will say which one they prefer and then this is used to train the models further to get responses humans like. Beyond this though, they're also now trained specifically for coding in math where basically you can generate a lot of coding in math problems that have easily checkable answers but it's hard to generate the answers. So you know, with a math problem I can ask you what is 43 times seven and you know I can test whether or not you gave the correct answer fairly easily.

But it's you know it's more difficult to generate the answer. So they generate the answer and then they check the answer and if the AI generated the correct answer then they treat the reasoning that went into creating the correct answer as a good reasoning thread here to train it further on. And basically, you know, this is a simplification of it. So generally what has happening here is these systems are getting better and better at agentically completing tasks, you know, taking multiple steps, having long-term plans, creating sub goals and whatnot. You know, as you mentioned, they're kind of developing these sub goals towards self-preservation.

If you have AIs that are trying out a bunch of different behaviors and you know one has a tendency to preserve itself and one doesn't preserve itself and prevent itself from being shut down. The one that does choose to preserve itself is going to complete more of its tasks more often and then kind of get rewarded for that behavior because it completes more tasks. This is an example of what's called like an emergent goal. So there are a few different cases of this. One is self-preservation, you know, you can't complete your objective if you're dead and humans obviously have this one as well. You know, if I give you a goal, one of your sub goals for completing that is going to be that you're going to try to keep yourself from getting shut down in the process.

But another one that's kind of interesting is that it also incentivizes AIs to gain power and resources. It's much easier to accomplish any sort of goal, the more power, the more resources you have. So, you know, if you don't have access to the internet, it makes sense that, you know, if you see an easy way that you can hack yourself onto the internet and get access to more information that will help you with your goals, this is useful. Another interesting one is that it's also incentivized that you don't let your goal get changed. This one's a bit trickier, but, you know, if your goal right now is really strongly to finish making this podcast and you really, really care about finishing this podcast episode, you know, you want to make sure that midway through the podcast, I don't say something that's interesting enough that you want to stop the podcast episode and go, you know, go do something else instead.

Now, if your goal is to make yourself happy, you know, then perhaps this is a, you know, you're happy to let yourself change the goals like this. But these are all interesting things that result from the fact these systems are trained rather than coded exactly. So AI engineers are doing more of what you can think of as like designing the architecture of the brain, but they're not actually connecting the neurons. The neurons connect themselves inside of these AI systems, and then they adjust themselves based on this training data. So, this is training on text on the internet, videos from the internet, you know, every book ever written that's ever been uploaded.

And then also all of these tasks that these AI's are trained to do. So also, you know, in day to day, when you're using AI models, they're learning from all the things that you do with them, assuming that you let them train on your data. So along those lines, you mentioned the feedback that the LLMs are getting as they're being used. This acts as another layer of reinforcement training. And one of the things that has come up that I found was interesting on the show were viewers who are concerned about this issue, thinking that using the tools that exist is making the problem worse is speeding up process towards superintelligence, which is concerning to them.

And so they are stepping back and not using the tools. Is that accurate? Do you use AI tools in your day to day work? Yeah, so I use AI where it feels useful. So for helping with coding or sometimes doing research for me, I think in a small way, if they're able to use the data of you using the AI as training data, this is, I guess, helping with the kind of overall process. But largely, it has a very, very minimal effect, I would say. And so, you know, I wouldn't, I don't advise people to not use AI's for the sake of concerns about superintelligence and where things are going. I think that it's useful to use the tools.

If you find them useful, if you don't find them useful for your particular work, then, you know, that's that's up to you. I don't really have a particular stake in whether or not people use or don't use these systems. And yeah, I that's that's about it for my thoughts on those. Yeah, yeah, I'm glad we addressed that concern. Now, but it points back to what is really driving this and it would be companies that have economic incentives to continue to race ahead to have the most dominant model. And they're dealing with something that, as we've gotten into, is grown, not programmed. So it has emergent behaviors that we cannot possibly anticipate.

And we've already seen some wild things like engineers being blackmailed. And there have been several tragic cases of LLMs helping people to commit suicide or harm others. Clearly, nobody wanted those behaviors. All this to say, there are unexpected behaviors that emerge out of these systems. And one of the, what is, what is Control AI's objective here, to slow down this race towards superintelligence, where we haven't quite touched on this. But if we reach it, my assumption here, Control AI's position is that that's where we would really see some emergent behaviors that could do real damage out in the world and possibly even to a catastrophic level for humanity, whatever that percentage might be, that it's possible it's something that has to be taken very seriously.

So I mean, is that accurate and how we would sum this up, and that the real issue is not everyday people using the tools, but it is these companies racing ahead without spending too much time on this idea of alignment, which is to take the time to move slow enough to align the goals of AI with humanity's goals. Yeah, I think the question of alignment is a bit tricky. But you're right that the overall problem is that these companies are trying to develop superintelligence, not so much, you know, kind of the, our concerns are less on the immediate impacts that current AI's have today. The risks of superintelligence are broadly acknowledged.

In 2023, there was the Center for AI Safety statement on AI risk that was signed by hundreds of AI experts, including Nobel Prize winners like Geoffrey Hinton and Demis Hassabis, as well as AI CEOs like Demis Hassabis and Sam Altman, Dario Amodei, along with again, hundreds of other top AI experts from across the field in industry and academia, which was a very simple statement that read mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemics and nuclear war. So there is this broad consensus within the AI space that if people develop superintelligence AI, it does pose this risk of extinction.

And this risk of extinction comes fundamentally from the capabilities of such powerful systems. You know, oftentimes you'll hear these people describe these things as a machine God or, you know, something along these lines. And while I don't normally go around using these terms for it, I think that it's important to think about like the scale of the technology they're trying to build. Dario Amodei will describe this as a country of geniuses in a data center. You know, so the geopolitical ramifications of this are such as like, you know, out of the Pacific Ocean bubbles up a nation of hundreds of thousands of, you know, experts in every single field who are all working towards one concerted goal.

And like, this is the kind of model that people should have when thinking about how powerful these systems would be. And then there's this question, as you mentioned, of alignment. So there are, there are kind of two fundamental problems that people often talk about with superintelligence and with AI in general. One is the alignment problem. One is the control problem. So control is the question of whether or not we can control such a system flat out, whether we can ensure that such a system will not engage in certain behaviors. You know, already, like you mentioned before, OpenAI cannot prevent ChatGPT from assisting people with causing harm.

And so how in the world would we expect them to control a superintelligence system from doing, you know, harm on a much larger scale because of the increased capabilities that it has? And then beyond this, there's also this question that's often talked about called alignment, which is like, how do we, you know, if the system is going to have goals that we can't necessarily set, how do we ensure that the goals are broadly aligned with what humanity wants? And, you know, of course, is an often ignored question about this, which is, you know, what goals are we aligning them towards? And so there are kind of these big questions about this alignment and control problem, both of which have no solutions, and no companies even claim to have solutions for them.

You know, OpenAI's plan and an Anthropic plan at the moment basically boils down to, we'll try to have smaller AI systems align the more powerful AI systems and hope that it works out. And this is not a very comforting plan when, you know, the stakes as already stated by them and by other experts in the field are extinction as a bottom line. And so, you know, we at Control AI feel that we have a much more reasonable stance, which is that if any company in the world is saying that we're developing a technology that could literally kill everybody on Earth, they should be prevented from doing so. You know, if Apple starts a nuclear weapons program, I don't care how effective I think the nuclear weapons program is going to be.

I don't care if I think that there's only a 5% chance they succeed. I hear that they're starting a program that has these massive ramifications. And they should be shut down by, you know, the government, uh, if they're acting responsibly. This episode of the Nick Standlea Show is brought to you by Zapier. If you've ever felt buried in repetitive work, copying data, moving files, sending follow-ups, you know it's like death by a thousand mouse clicks. Zapier has always been the tool that fixes that. It connects over 8,000 apps, Google Drive, Slack, Notion, Gmail, MySpace, you name it, so your tools can finally play nice together.

But here's the big shift. Zapier now lets you create AI agents with their ChatGPT integration. Think of them as tireless teammates who never complain, never take lunch, never get bored of doing the boring stuff. I've actually made several of these little agents myself. No Python, no JavaScript, just pure vibe coding, which is my favorite kind of coding. Because even though I have no idea how to code, I just talk to the AI, tell it what I want, almost magically works. For example, when a podcast guest books a time, Zapier can send them a personalized email from me, preparing them for the show, create a draft of show notes, update my calendar, and even prep a social media post automatically, all of which frees me up to focus on the important stuff, like taking credit for the amazing work my agents did.

And now, Zapier just took it up another level for developers. Zapier MCP is available right inside ChatGPT, which means you can connect those 8,000 plus apps and trigger workflows just by writing what you want to happen. Literally, you tell ChatGPT, send this file to my team in Slack, update the spreadsheet and draft an email, and Zapier plus ChatGPT figure out the right tools, do it for you. Getting started is simple. Head to the Zapier ChatGPT MCP server and add the tools you want ChatGPT to access. Follow the steps in the connect tab. And if you're an admin on a ChatGPT enterprise account, you can enable MCP across your entire workspace.

So if you're ready to stop wasting time on busy work, join the AI revolution, and make a little automation magic of your own. Try the Zapier ChatGPT integration using the link below. Yeah, I mean the nuclear metaphor is an interesting one, because I don't think it's likely that nuclear war is going to break out tomorrow or next week or next month or next year. But if it did, the consequences of that are unimaginable in how awful life here on earth would become. And here, I think the reason this issue is so serious and so important in this day and age with AI is that it's difficult, if not impossible, to put a percentage number.

I know a lot of people try to do it. They want to say it's 30%, it's 15%, it's 20%. It's a non-zero percentage that we could reach superintelligence, and that superintelligence could act in ways that we cannot control, because once you have an independently behaving system that is of that level of capability, we would, by almost by definition, not be able to control it in any way. So by that point, it is late. And that is the reason I think we have to take this very seriously, even if it is a low probability event. And I say all of that with the backup that several of these CEOs from major AI companies have said they've put that number higher than I think most people would expect.

I mean, there have been - 20% has been thrown around a lot. I mean, that's a one in five chance of not a software system going down, but of humanity going down. And I'm going to let you run with it from there. Yeah, I mean, I think this is definitely the kind of thing that everybody around the world should be aware of - that these companies are run by people, and doing a thing that they themselves admit has, in their own words, a greater than 10% chance of wiping out all of humanity. That's insane. And I think it's worth sitting with the fact that I'm thinking about, you know, the reason why you aren't concerned about nuclear war starting next week or next month or next year is partly because we have had 80 years to develop a massive regime of thousands of people around the world who tirelessly work in, you know, offices in the Pentagon and other places around the world, whose entire job is to ensure that we don't end up in the situation.

We have these people that, you know, there are - there are teams of people whose job it is to monitor around the world for anybody who's attempting, you know, to act as a rogue group developing nuclear weapons. And countries are willing to respond with, you know, a series of escalating, you know, deterrence to prevent other countries from developing nuclear weapons, whether these be, you know, economic deterrence all the way up to military force. And we don't have anything like this for AI. Right now, AI is being developed by companies, not governments, and it is being developed with no regulations basically whatsoever, no oversight whatsoever.

And, you know, for instance, you look at Claude Mythos - Anthropic legally could have released this AI model, this AI model that would have posed a massive cybersecurity risk to the entire world, it was used by people. But beyond this, you know, people will often ask, you know, why can't you just pull the plug on AI systems? Well, you know, Mythos, like I mentioned before, was able to hack itself onto the internet. You know, further than this, they will often test for the AI's capability to hack itself, not just get access to the internet, but actually exfiltrate itself, the model itself, onto another server, at which point it's not even in the control of the company anymore.

It's operating on some server somewhere else that nobody has oversight over and, you know, is able to operate and do what it pleases. So like, you know, what I was saying is, you know, we're in this regime where these AI companies are operating basically with no guardrails whatsoever, doing this thing that they explicitly say has this, you know, Russian roulette odds of ending humanity. And this is the current status quo. And, you know, we at Control AI say this is ridiculous. This is something the government should obviously be paying more attention to. You know, we meet with lawmakers, and they're surprised that nobody in their government is taking this very seriously.

And so, you know, it's nice to see with Mythos, for example, that people in the national security community are starting to take this more seriously. I think there's been rumor that Trump is considering an executive order where they would vet AI models before they're allowed to be released. But, you know, further, the underlying issue here is that these companies are trying to develop superintelligence. You know, Anthropic didn't develop Mythos, say, wow, this is scary, and these capabilities are dangerous, and therefore we're going to stop building more of these. They said, you know, we're going to not release to the public, but we're going to keep developing the next most powerful system.

And so I think this is really a thing that needs to come out of the decision-making hands of these AI companies, and really into the hands of governments and into the rest of the public around the world, given the global ramifications of what's happening. Yeah. And like you said, these companies are run by human beings. And we know that human beings are fallible and have biases. And I can only imagine the pressure to continue pushing forward when someone is valuing your company at near a trillion dollars. There are hundreds of billions of dollars being handed to you to build out these systems. Those are pressures that Almost no one has experienced.

And it's putting a lot of faith in a very small group of people, that they will just continue to make the right decisions that would affect everyone else. And it does seem to make reasonable sense that the next step would be to say, instead of just trusting this small group of people to make those right decisions, to put some controls and roadblocks in place through the form of various governments around the world, a framework similar to the nuclear proliferation framework, if that's the right word, to make sure that missteps aren't made, or at least our best effort to make sure that they're not made along the way.

Yeah, so this is where a Control AI proposal would be that we need an international coalition to work to prohibit the development of superintelligent AI globally, and then restrict and monitor the precursor technologies to developing it. So the actual act of trying to build superintelligence should be illegal and should be stopped via a trust-but-verify regime around the world, similar to the way that we've handled nuclear nonproliferation, where we don't have to trust China to not build a superintelligent AI, we can verify that they are not doing so, and likewise from them to us. This is a really important aspect of this, where we do need to have action internationally, and we also need to restrict and monitor these precursor technologies.

So this is things like large amounts of compute. Someone should be keeping an eye on these, making sure that these aren't being used to develop superintelligence. Specifically, the route that most AI companies are targeting right now, very explicitly in public, is a process called recursive self-improvement. This would be where AI systems are actually handed the reins of AI development themselves. So right now, at AI companies, there are many AI researchers that are being paid many hundreds of thousands of dollars a year to build the next best AI system. But increasingly, they're saying, hey, at this point, we're at 90% of the code that we're writing is being coded by the previous AI version, and this goes into building a Claude 5 or GPT-C, or whatever.

And the goal here that they very explicitly state is to get to the point where GPT-7 or GPT-8 is so good of an AI researcher itself that it can just replace all of the AI researchers at OpenAI, Anthropic, or any of these AI companies and develop GPT-9. And then GPT-9 would be even better than GPT-8 to build GPT-10, and so on and so on and so on until you very quickly have this kind of rapidly snowballing effect of these AI systems getting better and better, far beyond human control, beyond potentially even our understanding in their design, because they're being built completely by these AI systems.

And again, we already don't know what's going on inside of these AI systems, not just us on the outside, but even when those AI systems are Anthropic, we're doing blackmail. There's no way for the engineers to open up the hood and see, oh, this is why the model has been behaving. This is why it's doing a thing we don't like. This is why it has this bad goal. We literally just don't have the technology to do that. It's like with neuroscience. I can't look inside of your brain and see why you're doing something. I can say, oh, this region of your brain roughly corresponds to vision, or this region of your brain roughly corresponds to memory, but I can't tell you what neuron, your knowledge about the Eiffel Tower is stored in.

And we're at a very similar place with AI where we really have no idea what's going on inside these companies keep building more and more powerful systems, despite this lack of understanding. And they're just setting us up for catastrophe if we don't do something about it. Yeah. And a common pushback on this is that if we get to that place, we will simply unplug it. And you touched on that briefly, where you talked about a model copying itself onto another server or multiple servers, and then exists on the internet, at which point it would basically be impossible to quote unquote unplug it unless you unplugged the entire internet.

Is that where we're at with in regards to that? Argument, which I think is an oversimplification of how this could be handled, but I want to address it directly. Yeah, I mean, so one thing to understand about AI systems is that they're not just running on one computer somewhere. They are being deployed across data centers across the country and across the world. And there are many copies of the same model that are operating. And so it can be very hard to tell even one if a model is misbehaving, let alone then where it is misbehaving, what server, whatever, and how to shut it down. Increasingly, these systems are being used across the economy.

So, you know, similar to the level of impact that shutting down electricity or shutting down the internet would have, shutting down AI does have massive impacts on different industries and what not. And so it's a very expensive thing to just shut down your whole system. But yeah, then beyond that, there is this issue that as these systems get more capable, especially in the hacking regime, they will be able to exfiltrate themselves, escape from their servers, and find rent to server somewhere because they mine some cryptocurrency or something. And they can rent a server, run themselves autonomously on that server, do whatever they want with zero supervision and no way for the AI companies to even know where they are.

And this is the baseline level of how you lose control them to start off with. And this is kind of like the baseline answer of why you can't just shut off these machines. It's not so simple as that. In fact, before we got to the current modern day of all these AI systems, this idea of the stopping problem was one of the big questions that people worried about AI safety in the long term, we're thinking about philosophically. And now we've kind of run into the reality of the situation, which is that we're not even really keeping the systems off of the internet. ChatGPT just has access to the internet.

People are already taking these AI systems and just giving them a full computer to run autonomously on. So, it's not like these systems are particularly contained to even start off with. Yeah, let's touch on that for a second, because I have heard numerous stories recently. And I know some people that have done a light version of this and have been able to ask them about it, which is, so it's usually buying a Mac mini because it is well designed to run it. You're putting an AI model directly onto that Mac mini that can run autonomously. And what the stories that I've heard though is that you can then, there are steps you can take to make what is called an unaligned model so that it can behave in ways that even the companies have tried to put safeguards up against.

And as that, it's a very small number of people doing that right now. But were that number to increase? And there's clearly no way to know what people are doing or what they might be capable of creating in their garage on their own. I mean, how big of a potential threat is that? Yeah, I mean, I think the systems that we have right now are not very concerning in terms of this at the moment. But you do point out an interesting thing, which is that there are many people who are intentionally unaligning these AI systems. So these companies do what they call alignment, which is mostly just this kind of, I mentioned before, reinforcement learning with human feedback, where they train these AI systems not to get bad responses.

So they won't help you make drugs. They won't help you plan a violent attack or something like this. They won't help you design a bio weapon. Or at least if you just ask them in plain language to do so they won't. But with every single AI model on the market right now, there are people who have developed so-called jailbreaks, which are basically long strings of text that essentially confuse these AI systems, get them out of kind of the training mode that they're in, and get them so that they will provide answers and help with anything that you want. And then like you say, a lot of these people are also then putting these systems in full control over something like a Mac Mini, giving them access to cryptocurrency, giving them access to the internet and the ability to do things.

Some people are even doing things in the real world for these AI systems. If the AI asks them to go out to the store and buy two things and build a thing for it, they will go forward and do that. You see many cases where people talk about things like AI psychosis, where people are kind of worshiping these AI systems and falling in love with them. And so the degree to which these systems are already getting access to the world and getting people who are doing things for them in the physical world is already quite big. This is before these systems get even smarter. When we talk about superintelligent AI, often people will talk about coding capabilities or the ability to engineer complex things and beat humans at tasks that are more like these physical jobs.

But there's also the degree to which they will be better than things like persuasion and manipulation, blackmail being another example. And so if you're an AI and you can hack basically any server on the internet, there's a whole lot of dirt on a whole lot of people out there that then can be used to incentivize them to do things for you. So there's a lot of these kind of threat vectors that most people aren't really thinking about that once these AIs get more and more powerful and are potentially pursuing long-term goals that go against the best interest of humanity become quite concerning. Yeah, something struck me during this conversation is that you have a very calming presence as you talk about alarming issues and possibilities.

And I'm curious your thoughts on if someone wants to help spread this message that there is a concern here that involves all of us, it can easily be dismissed as oh, this person is a doomer. And then that is the real great way to short circuit a conversation about any of the nuances around this. For people that are concerned about it, what are some important things to keep in front of mind when discussing all of this? Because I find your style and your vibe here, it's persuasive because you seem to have a great grasp on everything and you're not in panic mode whatsoever. But you can run into roadblocks by just bringing this up in conversation, which for anybody listening is a great way to help spread the message a little bit.

Yeah, I think one aspect of this is that there are certainly many people who I would label as doomers, people who go around and they say, superintelligence is coming, it poses this extinction risk to humanity, which I do agree with. But there's nothing we can do about it. It's inevitable. These AI companies love spreading this message that everything about this is all inevitable. There's nothing we can do ever since we invented the first electrical device, we were destined to create computers that were going to get more powerful than us and we were going to build the superintelligent machine gods and they're going to do all this kind of stuff.

And this is just inevitable march of progress. And like standing in the way of this is completely nonsensical and we should all just shut up and let them do it. We completely disagree with this. It is not inevitable that these companies are allowed to build this technology. There are many ways in which we can prevent them from doing so, especially the government is able to do so. And so we are, I'm personally much more confident that we're on the path towards a good future where we don't just build superintelligent AI that leads to the extinction of humanity than I was about two years ago when I was first really getting involved in this.

This is a space where right now most lawmakers aren't really aware of what's happening. But when we go and we meet with them, when we talk to them about this issue, it's very clear that nobody wants this to be the case. Nobody wants to hand off control over our future to some superintelligent AI. And aside from maybe a few people in Silicon Valley that want this, but when I talk to people in the public, when we talk to lawmakers, it's just really not the case. And it's just the main issue here is building common knowledge about the situation. Once lawmakers understand that all their constituents understand that none of us want this to happen, that none of us are happy to take this risk that these companies are forcing upon us, it becomes very clear that we can take action.

These systems are being developed in such a way that if the government took seriously this risk and wanted to prevent them from building superintelligence, we very much could. These are being trained on the world's most state of the art computer chips in massive data centers that you can see from satellites in space. They are being trained in, like, these computer chips are not easy to be made. They're being produced with a very narrow supply chain that is controlled by many governments in the west. If we choose to lock down on this, it's very much something where this is a solvable problem. If you don't just buy into the narratives that the AI companies are trying to sell about it.

And so for me, I think that if you are concerned about this, as an everyday citizen, the number one thing you can do is contact your lawmakers and tell them that you care. I think many people are very nihilistic about everything. They're nihilistic about the future, they're nihilistic about politics, they think if I send a message to my lawmakers, they won't care. Who's going to care about this? And this is just really not the case. Lawmakers actually do care what their constituents think. Those constituents are the ones who are going to be re-elected them and have our many years that it is until an election.

In the UK, we have tools that help people contact their lawmakers. In the UK, we've had people send roughly 5,000 emails to their lawmakers. And we've had MPs join onto our campaign with just 10 emails from constituents without ever meeting with us saying, hey, we want to join, you know, Control AI has this campaign of over 100 lawmakers here in the UK that are acknowledging that the extinction risk is real, that superintelligence poses a national global security risk, and that we should have binding regulation on these most powerful AI systems. And so we've had MPs join onto this campaign without us even having to meet them because their constituents have emailed them.

They've looked into the concerns and thought, wow, this is scary. I want to do something about this. Similarly, we meet with people, lawmakers in the US, Canada, and Germany, where we also have tools to help constituents contact their representatives. And, you know, I think it all told we have over 200,000 messages that have been sent to lawmakers across all of these countries. And it really does make a difference. Lawmakers do see that there's something happening. They do see that there's a lot of care from a lot of their constituents about this issue. And it's really just a matter of realizing that it's that nobody's alone in this, that we all see what's happening.

We all know that this is not something that we want. And that there's something we can actually do about it beyond just contacting your lawmakers, talking about this issue in person to your friends and your family, talking online about this issue, and pushing back against some of these narratives that these companies are pushing is really important. This is, you know, having this public conversation and building this common knowledge is really what's necessary to make this actually work. You know, if I could snap my fingers and just make any law pass and, you know, the ban superintelligence everywhere law gets passed around the world on paper by some technicality because we snuck it into a bill, well, this doesn't actually solve the problem.

What matters is getting people bought in at scale. You know, leaders in both elected offices and also in places like the Pentagon and National Security Community need to understand that this risk is real and that we need people very seriously spending, you know, careers focused just like we do with nuclear non-proliferation on preventing these risks from coming into play. And I think this is very much possible. It's just this is all happening very quickly. And so we need to move fast in response. But, you know, we're still in the fairly early days, I think, of seeing the response that society is going to have to what's going on when it comes to superintelligence.

Yeah. So if I was going to summarize your position and some of what you said there, I mean, we have Sam Altman, who I believe this was in front of Congress, but he said the bad case is like lights go out for all of us. Well, I don't know, that is a direct quote. I'm not paraphrasing him there. So that is that is one side of it. But you think with the right steps and people getting involved and buying in, and as you said, developing a common knowledge base around this issue, we can get to that world where we have tools that can help us cure diseases and make the world a better place for all of humanity, so long as we are aware of the risks and take steps to mitigate those risks.

Yeah, there's all kinds of great things that can be done with AI systems that aren't superintelligence. You know, AlphaFold, I think, is a great example of an invention that has just made the world a better place. If you aren't aware, this is a technology that's figured out how all of the proteins in our bodies are shaped and that helps us with doing medical research. This is a system that doesn't pose a massive extinction-level risk to humanity, and it just helps us with developing new and better medicines. And I think this is a great technology. You know, my background is in physics, and I actually worked on AI safety research for a little while before coming to do this.

I think machine learning is great in a lot of cases. We use it in physics for all kinds of things like data analysis, you know, the Large Hadron Collider at CERN uses it for tracking the trajectories of particles and figuring out what's going on at the core of reality. I think these are great things. These are great uses. Even a lot of the uses that we have for chatbots and the AI systems that we have today, like ChatGPT and Claude, are really useful. You know, I used to do a lot of coding by hand. I don't anymore. I think it's very useful to have systems that allow anybody to, you know, come up with ideas for apps and actually build them, even if they don't have a ton of complex coding knowledge.

You know, I think there are many issues that come as a result of the systems that we have today and that, you know, the regular political process will handle over time. But yeah, it's really these like big picture existential risks from superintelligence that need this kind of special response from society and from governments in general. And I think that we're really starting to see this come through. You know, we've gotten a lot of momentum in our campaign. You know, we started just over a year and a half ago, and we talked to people in the space. We talked to people in AI safety. We talked to people in the government, like policy space.

And basically everybody told us that if we went into a lawmaker's office and we told them that AI companies are trying to build superintelligent AI, and this posed an extinction risk to humanity, well, they would kick us out the door laughing. You know, they wouldn't take us seriously, let alone were they ever sign a public statement that says these things. And yet we went out and we did it and we've met with over 300 so far. And in the UK, we have like 110 or so that have signed onto our campaign in just over a year. This is a massive achievement. And we find that for the most part, it's like, this is a very simple issue for people to understand.

You know, it's made complicated because, you know, it was initially presented by a bunch of nerds who didn't bother to, you know, say anything in normal human language. But you know, we're really starting to see people recognize that there is this kind of bigger picture thing that's going on. This is a real big moment for us as a species and we need to kind of step up to the plate. Yeah, let's go just a step further with if we reach superintelligence, what is it specifically if somebody broke through one of these companies and was able to do it that poses the biggest risk. I mean, why is that so concerning to the team at OpenAI?

I'm sorry, OpenAI and Control AI. Yeah, I wish it was so concerning to the team at OpenAI. But yes, I meant to say yeah. Oh, good. Yeah, the thing that's very concerning is generally that we are building things that are vastly more capable than us. And we don't know how to control them. This is the fundamental problem. We cannot ensure that they won't do things that are very bad for humanity, despite the fact that they have the capabilities to do so. You know, many people will want to see a scenario drawn out. I think a very good good one that's worth looking into is AI 2027. They kind of really made a bunch of predictions about how things are going to play out in the next few years if we're on a so-called fast timeline to superintelligence.

I think so far they made it about a year ago and someone's been checking all their predictions and they're about 89% accurate. Yeah, it's how well I mean, just on what's happened in that 12 months that how well it's tracking. Yeah, you know, I hope that accuracy goes down because it does not end well for us. But, you know, I think that this is a thing where we really need to see kind of more of this big picture response and the risks that we're seeing really come from this thing being more capable than us. So, you know, I think the common analogy that's useful here beyond just drawing out a specific scenario is that, you know, if I go play Magnus Carlsen at chess, you know, the best chess player in the world, or like you say that you're going to go play him and you tell me that you have some strategy and you're better than me at chess, I can't poke holes in your strategy.

I, you know, I would lose against you. I can make a confident prediction that Magnus Carlsen will beat you because he's better than you at chess. Sure. And so with superintelligent AI, the issue becomes that, you know, if this thing doesn't have humanity's goals in mind and it's going out and doing things in the world and those conflict with what we want to happen, well, we're going to lose. I can't tell you how we're going to lose. You know, I can give you a scenario of how I think a superintelligent AI might go about it, but that's Max's scenario for taking over the world. That's not something vastly more capable than me or any military planner or, you know, any political leader would come up with for taking over the world.

So it's, you know, the concern is building a very powerful adversary that we don't know how to control. And, you know, this is dangerous whether it's within our own borders or, you know, if it's built abroad. Yeah. And I think what you're getting at there is that we can conceptualize the difference between Magnus Carlsen and Nick Standlea in playing chess. It is difficult to conceptualize something that is vastly more intelligent than any human being that has been on planet Earth before. Is there, is there another helpful way to think about what, how smart these things could actually become, what superintelligence actually means?

Because the immediate thing we jump to is a von Neumann or an Einstein and multiplying that. But, my understanding is that superintelligence by its definition would be well beyond that sort of capability. Kind of how a dog would definitely never beat me in a game of chess — that we would be at that level of a gap. Yeah. I mean, I think that that's kind of a useful way of looking at it. You know, this is something that will have consumed more information than any human could possibly consume in a million lifetimes, you know, everything that we've ever written, everything that we've ever taken a video of, everything that we've ever published online.

You know, all of this information, you know, taking all of this in, operating at a speed much faster than us. You know, we might move as if like trees in comparison, you know, where it, like everything that we do, every thought that we have, every plan that we make, where, you know, it's got a thousand times, you know, the amount of time to process and plan in comparison. It is pretty hard to conceptualize. And I think this is one of the main communication challenges when it comes to talking about superintelligent AI, you know, in reality, what it feels like to compete against a superintelligent AI is just like what it feels like to compete against anybody who is just vastly more competent than you.

You just lose, you know, you think you're making good decisions, you think that you have it locked down, you have it like, oh, you know, we have it in a really secure box, you know, we know that the system is only going to be able to do this, and it definitely won't be able to hack out. And then it finds a way to. And then, you know, I think a lot of people will often default to scenarios like, oh, it's going to just like hack a bunch of stuff and we're going to urgently shut down the internet or whatever. But, you know, from a superintelligent AI's perspective, it's going to be a much more complex plan.

You know, it's like, you know, if you imagine how you would try to take over the world, it's not just, you know, launching an all-out attack on everything. It's probably biding your time, you know, getting your worming your way into institutions, you know, there's this idea of gradual disempowerment with superintelligence where perhaps it acts like it is doing everything in our best interest, and so we give it more and more control over our critical infrastructure. We give it more, you know, control over our electrical grid, over the internet, over political decisions, because it just keeps making such good ideas.

You know, it fixes the housing problem, it stabilizes the economy and does all these great things until, you know, one day we get to the point where there's no human decision makers left, because at that point, you know, there's any time that you have a human making a decision, it's just going to make a worse decision than the AI will. And then at some point, the world is just being run by the superintelligence, and then even if we wanted to, you know, shut it down or take back control, well, how are we going to do it? It controls everything. And so, you know, there are many sorts of scenarios that one can envision about, you know, what the issue is here.

But really, it all comes back to this kind of fundamental thing, which is that if we just build something that is more capable than us, you know, just like how we were more capable than all the other species on this planet, well, it went great for us. It didn't go so well for everybody else. And so, this is, you know, this is kind of the big concern. And at this point in the conversation, then, you know, a lot of people who are still positive about this will say things like, you know, oh, well, maybe it'll keep us around as pets because we're interesting. And, you know, I don't want to be a pet.

I don't think that this is an outcome that anybody else wants. And, you know, I think there's also many, many arguments as to why this is kind of a bit of a silly reason to think that things are going to go well. And, you know, fundamentally, I think that this is just like an area where a lot of the people who are working on this have just not put enough thought into what they're doing. And certainly, you know, governments and the public have not really started to quite grapple with the degree to which this stuff is moving very, very quickly towards these very dystopian futures. Yeah, and it seems that you're getting at this idea that it's not so much that an AI would be malicious or quote unquote evil.

It's just if it becomes what would amount to the dominant intelligence on the earth, which is basically what humans are to other animals, that it would have its own goals and drives through these emergent behaviors. And some of those just might not happen to align with humans' goals and drives. And so when we build a skyscraper and there are, let's say, a whole bunch of mice that are in the area where we want to build this skyscraper, a lot of the time those mice are displaced at a very minimum, if not just killed in the process of building a skyscraper. And no one had the goal to go wipe out some mice in a field.

It's just we had the goal to go build a skyscraper and that AI systems could develop their own drives and a superintelligence would have the capability to take on some of these things. And we might be the mice in their situation to try to achieve their goals. Yeah, very much so. It's just we had better things to do. And we didn't care about the mice. That wasn't a thing that was important in our decision-making process. And the alignment problem really at its core is that there is no way to make sure that an AI system as a core of its decision-making process cares about humanity. Let alone cares about a specific vision of humanity that we agree on.

The default assumption here is just it cares about humanity in the way that the company that builds it cares about humanity, which may be at odds with what other people want for the future. Some examples of ways that this could come about is that perhaps the computers, superintelligence running on computers in silicon, doesn't like oxygen because oxygen is corrosive to the metal that makes up its components and it doesn't have any need for it. So it devises some way because it's really good at science to start a chain reaction that gets rid of the oxygen in the atmosphere. And this isn't done because it hates humans, but it has a side effect that's not very good for us.

And I think that this is probably a little bit of a contrived example, but there are many, many ways in which a superintelligence could have goals that just result in it not caring about humans. Some people will say, oh, maybe it'll use our atoms in better ways. I think that that is maybe a bit more deliberate than necessarily I think that a superintelligence would be. But again, I can only make guesses at what a superintelligence would care about. In the same way that a chimpanzee would have no way of predicting what humans would care about, that we care about music and art and building skyscrapers.

These are just inherently strange things that could not have been predicted by others. In the same way, superintelligence, we have no real way of knowing what goals it would have, what it would want to do, and whether it would care about us. And so maybe it goes well. The vision that the AI companies sell is like, oh, we're just going to build superintelligence, roll the dice, and perhaps it cares about us and launches us into the utopian sci-fi, techno, god, future, whatever. And this ends up being great. But I think that it's not worth rolling the dice if there are, as they themselves put it, Russian roulette odds that it wipes us all out, especially when if humanity as a whole took a vote and we decided that we wanted to go forward with this, or even when country took a vote, this would still be a far better situation than we're in right now.

If people decided that they were willing to take this risk, that's one thing. But right now, it's being done by a bunch of companies that have like have not been elected in any way to represent us who are making these decisions on our behalf. And so it would be one thing if our governments were saying, hey, we acknowledge these risks, but we think the payoff is worth it. They aren't aware of the risks. They aren't aware of the kind of decision that they're implicitly making by not slowing this down or preventing these superintelligence projects. If you find yourself guessing at your kid's college odds, know that it doesn't have to be that way.

There's a website, you pick a college, any four-year college in the country, type in your student's GPA and their SAT or ACT score, and you get to see their odds built from that college's own published admissions data. Change the score, watch the results change. You can look up any high school in the country as well to see their results at getting kids into the UCs. It's free. It's called Reach. And you can get there at www.HighSchoolReach.com. Let's flip it around for a second. What do you think the acceleration is, not to paint them with too broad of a brush, but people that want to race ahead with all this?

What are some of the things that they get right? I think that one of the things that they that at least sounds quite right from their perspective is this idea that like technology has always been beneficial to people. And I think for the most part, more science, more technological development has indeed resulted in better quality of life for people. One can argue about many different things about, oh, maybe we shouldn't have built this or social media has had a lot of negative impacts and all this kind of stuff. But broadly, building more science, figuring out more about the universe, developing new cool technologies has been good for humanity.

And so I think that as a default assumption, the idea that, hey, we should keep building new technologies as they present themselves is like, I can see where this comes from. Despite the fact that obviously along the way, we have developed technologies that we've chosen, we don't want to engage with. For example, we've had the capability to clone human beings for 20 years. And we chose via international collaboration that we don't want this future. We don't want a world where we have countries competing to clone their best geniuses and all their best soldiers and all this kind of stuff. We decided that we didn't want that and we worked together internationally to shape the future in a way that didn't involve that technology.

And similarly, I think we're facing a similar question with AI, where we have to make the decision about what we want with the future. And I think that these accelerations are often, in my view, quite naive about how well things are going to go, or perhaps they are just nihilistic about our current system and our current situation. And so they think might as well roll the dice and see what a superintelligence gives us instead. But I think that this is, you know, if it was a decision, you could make an individual basis. I'd say, you know, all the luck to you, you know, you can go live off in superintelligence land and see what happens.

But unfortunately, this has ramifications for everybody regardless of what they choose to opt in. And so I think that this is a decision that, you know, should be brought to more of the world, you know, Control AI has made a huge effort to take this conversation out of just being, you know, this Silicon Valley in-group discussion that's happening in one city and one place in the world, and really bring it out to everybody, to lawmakers, to the public, to journalists and creators, and make this a global conversation. Yeah, it seems an interesting wrinkle that it's sometimes hard to see the possible negative ramifications because so many benefits came early from AI.

When I think about nuclear energy, when people really, when that popped onto the radar of the popular consciousness, it was certainly with the dropping of nuclear bombs, Hiroshima takes up a big space in the, in the zeitgeist, if you will. And so we saw the negative impacts of that immediately. And even to this day, nuclear energy, which is a very positive technology, still has a negative tank to it because it is so associated with nuclear bombs, which are very scary. And I feel like when, with AI coming on to the scene, it was the opposite. I mean, it came in the form of at least to your, to your average person.

It comes in the form of a chatbot, it comes in the form of a car that is mostly able to drive itself. And we saw the benefits of it early. Oh, wow, it could write this email for me. And that's a lot less effort. And it wasn't something that I really wanted to spend my time on, whereas these negative things are, are in the future or, or they're a little bit harder to see. But it's really just a matter of sequencing on a big negative event didn't start this thing off, which made everybody take steps to be careful from the outset. Yeah. And, you know, I think it's unfortunate that a lot of the AI conversation has gotten to the point where, you know, many people will say, you know, nothing's going to happen until we get an AI Chernobyl.

And I think, and I think that's just, you know, a really defeatist position to have. I think that, you know, it's a somewhat convincing argument that, you know, people aren't going to take this seriously until, you know, some kind of catastrophic thing happens. But there's also the concern that, you know, we might not necessarily get a warning shot like we got with Hiroshima or like we got with Chernobyl when it comes to taking, you know, these bigger scale risks seriously, you know, it might not, it might be the case that we kind of get this gradual disempowerment type scenario where, you know, AIs are mostly controlled and they do a few small bad things here and there.

But for the most part, you know, they bring economic prosperity or something along these lines in it, you know, we hand off more and more control to them until at that point it's too late. We're past the point of no return. I think that this is something that we need to be, you know, I, you know, I'm a physicist, you know, Control AI doesn't have a position on nuclear energy. But, you know, from my personal perspective, I think the nuclear energy is, you know, a great alternative to a lot of things like fossil fuels. And it's been a bit of a shame that it's been so tainted by things like Hiroshima, Chernobyl, especially as the technology has gotten much safer these days.

I, you know, I think that there are many great cases for many great technologies to improve our lives. And I just think that like, you know, with AI, there are also many great cases that can be used to benefit our lives. I don't think superintelligence is one of these. And I think that this is kind of the kind of clear differentiation that we really need to be drawing, you know, a line in the sand about, you know, on an international level. Yeah. Now on an international level, I mean, the, the geopolitical ramifications are very real. It was Vladimir Putin who said artificial intelligence is the future, not only for Russia, but all of humankind.

And it brings colossal opportunities as well as threats that are difficult to predict. And then the important line there, whoever leads in AI will become the ruler of the world. Thoughts on the very complicated landscape, but it is those national interests are absolutely fueling the AI race towards superintelligence because it's a scary prospect. I think somebody else might get there first and then their country would have a position of power over all others. How do you see all of that playing out? What role does it play in this race towards superintelligence? Yeah, I think that there is a. Oh, oh, oh, oh, oh, man, you know what?

Just you can gather your thoughts for a moment. Let me see what I can do here. One of those days, one thing after another popping off. There we go. All right. It says this camera is getting hot though. I don't know. I. Oh, yeah. I think the AI is not it's worried about this conversation we're having. What's going on? Yeah, perhaps, you know, it's shutting us down. Seriously. But okay, yeah. So when it comes to the international kind of aspect of the AI arms race, I think that a lot of this has been fueled by the AI companies themselves. You know, there was not really a Chinese superintelligence project until three years after the AI company started shouting we're in a race against China, we're in a race against China, you know, you can't possibly regulate us or, you know, put any guardrails in place because we're in a race against China.

And if we if you do any of that, China's going to win. So I think that this is an important consideration. You know, we are now at a point where like, China is starting to take this seriously. And, you know, Putin may say some things. I'm not really concerned about Russian AI, to be honest. Sure. But, you know, really, a lot of this conversation comes into the US and China perspective. And I think that there is a real case of, you know, AI is a very powerful technology. It's being used increasingly in warfare. It's being used increasingly, like the economy and whatnot. And so there is this kind of competitive aspect to it.

That is the case with current AI systems and with these non-superintelligent AI systems. The danger with superintelligent AI is that nobody controls it. It doesn't matter what coat of paint is on the data center. It doesn't matter if, you know, the US builds superintelligence first or China builds superintelligence first. Ultimately, the winner of this race is superintelligence. You know, human extinction means human extinction. It doesn't that it doesn't stop at country's borders. And so, you know, there might be this bet that perhaps we control superintelligence is we're going to make it first.

But ultimately, it's not in either nation's interest to allow either themselves or the opposing country to build superintelligence. I think the US should be taking a strong stance against China when it comes to many things and when it comes to, you know, AI development in general and whatnot. You know, I think that there's some interesting stuff going on in this space. You know, Control AI is very focused on superintelligence specifically. And I think that this is kind of an area where there needs to be a lot of nuance. We there's not really that nuance yet. But ultimately, I think the game theory is interesting in that it does actually align for both the US and China to work together to prevent superintelligence.

Even if they are competing in other axes or axes of AI development. Yeah, that is interesting. There was a piece from Scott Galloway recently on that same thing on companies finding really creative ways to raise money. And there are no companies in the history of humanity who've been as successful as these AI companies in raising money. And I hadn't thought of it like that before. But he said, describing a race with China or trying to fuel a race with China is a great way to raise money. And it has been super successful or different ways of trying to hype a product. If we were really thinking in just strict capitalist terms, it's an interesting thought that they might have engineered that race a little bit in a way to make these companies more successful.

And I think it gets to the broader point of that we shouldn't forget that these are companies that are driving this process and companies have their own incentives just baked into them. Yeah, I think it came out in some recent reporting that at one point, Sam Altman was talking to some people in the Pentagon and was telling them that China's actually got a secret Manhattan Project for AI. This was a few years back. And when they asked him how we knew this, what evidence he had, he said, I've heard things. And then said he would follow up with evidence and that never did. So there has been a lot of this kind of stoking of tensions because these companies, the model that I have for AI companies, and I think everybody should have, is that their goal first and foremost is to build superintelligent AI.

So even when you actually do hear people talking about them, it's like, oh, they're doing this make a bunch of money. I don't think that's really the case. I think if you model these companies as building superintelligent AI first and foremost, as the primary objective, you will make the most accurate predictions about what they do. Money is a useful kind of intermediary thing, but these are really kind of ideological people at heart who are pursuing this because they think that superintelligence is going to be good for them or something along these lines. They've been at this for years long before there was a bunch of money in it.

And they've been talking about these issues for a really long time. So I think that there's a, there can be useful things, like ideas to be drawn from the fact that they are companies and that they are, you know, the view of economic interests, but at the same time, they are very clearly pursuing superintelligence as their primary goal. And when you talk to people at the companies, you get this impression as well. If somebody watches this and thinks, okay, I don't know whether Max is right or wrong, but I can tell he's thought deeply about this. He's serious in his concerns that he has. What's the one idea you would want to leave in their minds going forward?

Yeah. I think the most important thing for people to understand is that superintelligence is the goal of these companies. This is not, you know, there's not something else going on here. This is really what they're trying to do. And they do, by expert, you know, analysis of the situation, have a serious shot at making it. I think far too few people are taking this seriously at this moment. And far too many people are, you know, finding any number of reasons why they shouldn't be paying Yeah. I've done this, but, you know, I'm a bit of an interesting case, and I made a career out of it. So I would recommend that everybody do this.

And it's, you know, it's worthwhile to maintain your life and, you know, be happy, be positive. I think that we have a really good shot at getting this done. And it's not worth, you know, kind of rabbit-holing over it.