Can a treaty actually stop the race to superintelligence?

It is a drafted international agreement, written by MIRI's governance team, aimed at preventing the premature creation of superintelligence. Bourgon describes it as non-proliferation in style: consolidate AI chips into monitored data centers, cap training compute, monitor training runs below the cap, and restrict certain capability research. He says it would likely be paired with a centralized effort to solve the underlying scientific problems, so the restrictions do not have to last forever.

MIRI has written more than one framing. One is a multilateral version built around international institutions. The one Bourgon says they talk about most at the moment, given the current administration's perspective, is a bilateral agreement between the United States and China that then forms a coalition of other countries around it.

Bourgon says the agreement sets a cap of 10 to the 24 floating point operations for any single training run, which he acknowledges is a fairly low bound since models are already trained above it. Below that, a narrower band from 10 to the 22 up to 10 to the 24 would be monitored with verification and enforcement mechanisms. The cap is set low on purpose, because algorithmic efficiency keeps making it possible to reach the same capability with less compute.

No, and Bourgon addresses this objection directly. The agreement names an amount of compute an individual can hold with no monitoring, which he puts at around 500,000 dollars worth of chips, described as 16 H100 equivalents when the proposal was written. He says the restrictions target a very specific class of activity with a fairly large collection of chips, not consumer hardware.

Bourgon's argument is that the inputs are unusually easy to get a grip on. Training a frontier model currently needs data centers with more than 100,000 chips running for months on the power of a small city, only about three companies design those chips, roughly 90 percent are manufactured at TSMC in Taiwan, and only ASML in the Netherlands makes the machines that manufacture them. Finding the majority of those chips and putting them under a monitoring regime would be a lot of work, he says, but far less than World War Two levels of effort.

Nick puts that objection to him directly. Bourgon does not dismiss it. He says the thresholds are set so that a hidden actor should not be able to assemble enough compute for dangerous training, that some of the research restrictions exist specifically to raise the cost of going underground, and that MIRI recently updated the agreement after running research on distributed training trends. He also says dark data centers are a real worry, while adding that very large data centers are hard to hide and intelligence agencies are good at finding them.

Partly, and Bourgon names the ways it is not. With nuclear weapons the danger was demonstrated immediately and everyone agreed on the threat, whereas with AI there is no superintelligence sitting there for people to be scared of, and by the time there is, he says, you are already in trouble. Nuclear also lacked the enormous economic pull that AI has. The strong analogy, in his view, is controlling the physical inputs, and he argues the AI inputs are actually more concentrated than uranium and centrifuges.

No. Bourgon says the restriction targets one narrow thing, the race toward ever more generally capable frontier systems, while trying to keep space for everything else people want to do with AI. He does not undersell the cost, saying there will probably be useful systems that never get trained under such a regime, and that taking all the chips into a monitoring scheme is a significant intervention.

He points to the 1950s, when plenty of smart people thought nuclear proliferation and another world war were simply inevitable, and it did not play out that way. He tells the story of Reagan watching the TV film The Day After, being shaken by it, and later meeting Gorbachev. His read on policymakers is similar: most have not engaged deeply because they are busy, and when they do, many of them find the concerns to be common sense.

It is his reframe of the doomer versus accelerationist split. On his account, the accelerationist position holds that the race is inevitable and the only way out is through, so you let it rip and hope problems get fixed along the way. The people labeled doomers, he says, believe the world can actually rise to the occasion and govern this, which he considers the more optimistic of the two beliefs.

Because Bourgon says nobody can predict where the threshold is. He cites a Wright brother saying heavier than air flight was a thousand years off two years before doing it, and Rutherford dismissing atomic power the day before Leo Szilard worked it out. He also points to a cyber capable model Anthropic released privately, which he says jumped in one generation from mildly helpful to better than most security professionals at finding software vulnerabilities. His answer to when we should stop is as soon as possible, and he says he does not love that answer.

Bourgon says the companies are openly aiming at AI systems that can do the research and development of AI itself, and that they talk about one to three years to reach it. What worries him is that this is an internal deployment question, not a product launch, so there is no responsible disclosure moment and outsiders might not see it happen. He says he wishes governments were tracking progress toward it much more closely.