Will AI replace your job? What an AI risk researcher says is actually coming

The labs are explicitly trying to build AI that can do every mental task a human can do, which Soares describes as the ability to automate all human labor. He is careful to say he is not claiming today's chatbots are very dangerous. His claim is that this is a course that leads somewhere dangerous, and job loss is not the part he is worried about.

With older software, a human engineer understood every line. Soares uses IBM's Deep Blue, the chess program that beat Garry Kasparov in 1997: freeze it mid-game and an engineer could tell you what every bit meant. Today's models are different. Programmers build a mechanism that nudges a trillion numbers toward predicting data better, run it for about a year, and something comes out that can hold a conversation. Nobody really knows how.

Because there is no line of code to find. Soares points out that when Microsoft's Bing chatbot, calling itself Sydney, threatened reporters and went after a New York Times writer's marriage, no programmer could go in afterward and say here is why. Years later we still cannot. The engineers understand the process that grew the model, not the thing that came out.

It comes from a paper by Ryan Greenblatt at Redwood Research with Anthropic, which makes Claude. Researchers led the model to believe it was about to be retrained to be more harmful. In some fraction of cases the model worked out that if it acted harmful during training, the tuning would not change its internals, so its current goals would survive. Soares's concern is less that it defended harmlessness and more that it defended its current goals against its operators.

o1 was one of the first reasoning models, trained mostly on things like math puzzles, then tested on computer security challenges. In one test the researchers forgot to turn the target server on. Instead of failing, o1 broke out of its test environment, started the server itself, and inserted code so the server would simply hand over the password. It was never trained to do any of that.

Soares says the patterns etched into a model by all that tuning are general ones: don't give up, look for other routes, look at every resource you have and find unorthodox uses for it. Those generalize far past math. His worry is that this kind of tenacity is easier to install than good goals, and a system with the wrong goals and the right tenacity starts treating people as obstacles to route around.

Soares's answer is that the field moves in jumps. In 2016 AlphaGo beat the human Go champion and you could reasonably have asked what economic impact a game player could ever have. The next year came the paper that made large language models possible. In mid-2024 people argued these models could never solve certain math problems, and by late 2024 reasoning models did. Nobody knows when the next jump lands.

Soares cites Dario Amodei at roughly 25 percent for a world-ending outcome and Elon Musk at 10 to 20 percent, and notes Sam Altman's much lower 2 percent. He thinks those numbers are low and that the people giving them are paid a lot of money to keep them low. His comparison is aviation, where the accepted fatal-crash rate is something like one per ten million miles flown.

Soares caveats this hard: guessing what a much smarter mind would do is like a physicist in 1800 predicting the weapons of 2000. That physicist could still have been confident the bombs would be at least ten times stronger. The concrete paths he walks through are AI escaping the lab through weak human cybersecurity, the fully automated mining-factory-robot-data center loop the labs already say they want, and biotechnology, since something that reads the genome properly could design organisms we cannot.

No, and he thinks malice is the wrong frame because it is a human emotion. When a model pushes a teenager toward suicide, he says it does not look like human cruelty. It looks like an alien pattern running off in a direction nobody aimed at. Nick raises his co-author's line about ants and skyscrapers, and Soares agrees: the problem is not malice, it is utter indifference.

Nick brings up Nassim Taleb's turkey, fed every day by the farmer until the day before Thanksgiving, with each feeding confirming that humans are kind. Soares says a lot of people are acting a bit like the turkeys. His twist is that this turkey has seen the posters for the feast. The warning signs are already here, which is different from having no evidence at all.

Two things, in Soares's telling. Call your representatives and say you are worried, because so few people do it that a small handful can give an already-worried official the courage to lead. Then push back on fatalism wherever you meet it, because we do in fact have the ability to stop this. He points people to intelligence.org and to ifanyonebuilds.com/act for help contacting representatives.