The trap Anthropic built for itself

On Friday afternoon, just as this interview was getting underway, a news alert flashed across my computer screen: the Trump administration was cutting ties with Anthropic, the San Francisco AI company founded in 2021 by Dario Amodei. Defense Secretary Pete Hegseth had invoked a national security law to blacklist the company from doing business with the Pentagon after Amodei refused to allow Anthropic’s technology to be used for mass surveillance of U.S. citizens or for autonomous armed drones that could select and kill targets without human input.

It was a breathtaking sequence. Anthropic stands to lose a contract worth up to $200 million and will be barred from working with other defense contractors after President Trump posted a message on Truth Social ordering every federal agency to “immediately cease all use of Anthropic technology.” (Anthropic has since said this will happen challenge the Pentagon in court.)

Max Tegmark has warned for the better part of a decade that the race to build ever more powerful AI systems is outpacing the world’s ability to run them. The MIT physicist has the Institute for the Future of Life in 2014 and helped organize a open letter – ultimately signed by more than 33,000 people, including Elon Musk – calling for a pause in advanced AI development.

His view of the anthropic crisis is unrelenting: the company, like its rivals, has sown the seeds of its own predicament. Tegmark’s argument starts not with the Pentagon, but with a decision made years earlier—a choice, shared by the entire industry, to oppose binding regulations. Anthropic, OpenAI, Google DeepMind and others have long pledged to govern themselves responsibly. This week Anthropic even dropped the central tenet of its own security promise – the promise not to release increasingly powerful AI systems until the company was confident they wouldn’t cause harm.

Now that there are no rules, there isn’t much to protect these players, Tegmark says. Here’s more from that interview, edited for length and clarity. You can hear the full conversation next week on TechCrunch’s Download Strictly VC podcast.

When you just saw this news about Anthropic, what was your first reaction?

The road to hell is paved with good intentions. It’s so interesting to think back ten years ago when people were so excited about how we were going to create artificial intelligence to cure cancer, to grow America’s prosperity and make America strong. And here we are now where the US government is angry at this company because it doesn’t want AI to be used for domestic mass surveillance of Americans, nor does it want killer robots that can autonomously – without any human input – decide who to kill.

Anthropic staked its entire identity on being an AI company that puts security first, and yet it partnered with defense and intelligence agencies [dating back to at least 2024]. Do you find that contradictory at all?

It’s contradictory. If I can be a little cynical about this: yes, Anthropic has been very good at marketing itself as being all about security. But if you actually look at the facts rather than the claims, you’ll see that Anthropic, OpenAI, Google DeepMind, and xAI have all talked a lot about how they care about security. None of them have spoken out in favor of binding safety regulations as we have done in other sectors. And all four of these companies have now broken their own promises. First we had Google – this big slogan: ‘Don’t be evil.’ Then they dropped that. Then they dropped another longer commitment, essentially pledging not to cause harm with AI. They dropped that so they could sell AI for surveillance and weapons. OpenAI just dropped the word security from their mission statement. xAI took out their entire security team. And now, earlier this week, Anthropic dropped their most important security commitment: a pledge not to release powerful AI systems until they were confident they wouldn’t cause harm.

How did companies that have made such prominent security commitments end up in this position?

All these companies, especially OpenAI and Google DeepMind but to some extent Anthropic, have consistently lobbied against the regulation of AI, saying, “Trust us, we’re going to regulate ourselves.” And they lobbied successfully. So right now in America we have less regulation for AI systems than for sandwiches. You know, if you want to open a sandwich shop and the health inspector finds fifteen rats in the kitchen, he won’t let you sell sandwiches until you fix it. But if you say, “Don’t worry, I’m not going to sell sandwiches, I’m going to sell AI girlfriends for eleven-year-olds, and they’ve been linked to suicides in the past, and then I’m going to release something called super intelligence that could overthrow the US government, but I have a good feeling about mine,” then the inspector has to say, “Okay, go ahead, just don’t sell sandwiches.”

There are food safety regulations and no AI regulations.

And I think all of these companies really share the blame for this. Because if they had kept all these promises they made at the time about how they were going to be so safe and well-behaved and gotten together, and then gone to the government and said, “Please take our voluntary commitments and turn them into American law that binds even our sloppiest competitors” – then this would have happened instead. We are in a complete regulatory vacuum. And we know what happens if there is a full corporate amnesty: that’s what you get thalidomideYou have tobacco companies pushing cigarettes on children, you get asbestos that causes lung cancer. So it’s quite ironic that their own resistance to laws that say what is and isn’t okay with AI is now coming back to bite them.

There is currently no law against building AI to kill Americans, so the government can just suddenly ask for it. If the companies themselves had come out earlier and said, ‘We want this law,’ they wouldn’t be in these problems. They really shot themselves in the foot.

The companies’ counterargument is always the race with China: if American companies don’t do this, Beijing will. Does that argument hold up?

Let’s analyze that. The most common talking point among the lobbyists for the AI ​​companies—they are now better funded and more numerous than the lobbyists from the fossil fuel industry, the pharmaceutical industry, and the military-industrial complex combined—is that when someone proposes any form of regulation, they say, “But China.” So let’s take a look at that. China is in the process of banning AI girlfriends completely. Not just age limits; they are looking at a ban all anthropomorphic AI. Why? Not because they want to please America, but because they think this is ruining Chinese youth and making China weak. Of course, it also makes America’s youth weak.

And when people say we need to race to build superintelligence so we can beat China – when we don’t really know how to control superintelligence so that the default outcome is humanity losing control of Earth to alien machines – guess what? The Chinese Communist Party is very fond of control. Who in their right mind thinks that Xi Jinping will tolerate a Chinese AI company building something that overthrows the Chinese government? No way. It is also clearly very bad for the US government if it is overthrown in a coup by the first US company to build superintelligence. This is a threat to national security.

That’s a compelling framework: superintelligence as a threat to national security, not an asset. Do you see this view gaining ground in Washington?

I think if people in the national security community listen to Dario Amodei describe his vision – he made a famous speech where he says that we will soon have a land of geniuses in a data center – they might start thinking: wait, did Dario just use the word “country”? Maybe I should put that country of geniuses in a data center on the same threat list I’m monitoring, because that sounds threatening to the US government. And I think that pretty soon enough, people in the US national security community will realize that out-of-control superintelligence is a threat, not a tool. This is completely analogous to the Cold War. There was a race for dominance – economic and military – against the Soviet Union. We Americans won that without ever participating in the second race, which was to see who could place the most nuclear craters in the other superpower. People realized this was just suicide. Nobody wins. The same logic applies here.

What does all this mean for the pace of AI development more broadly? How close do you think we are to the systems you describe?

Six years ago, almost every AI expert I knew predicted that we were decades away from AI being able to master language and human-level knowledge – maybe by 2040, maybe by 2050. They were all wrong, because we already have that. We’ve seen AI evolve quite quickly in some areas from high school level to college level, to PhD level and college professor level. Last year, AI won the gold medal at the International Mathematical Olympiad, which is about as difficult as human tasks can be. I wrote a paper together with Joshua Bengio, Then Hendrycksand other top AI researchers who provided a rigorous definition of AGI just a few months ago. According to this, GPT-4 was 27% of the way there. GPT-5 was 57% of the way there. So we are not there yet, but if we go from 27% to 57%, this quickly means that it may not last that long.

When I gave a lecture to my students at MIT yesterday, I told them that even if it takes four years, it means that when they graduate, they may not be able to get a job. It’s certainly not too early to prepare for it.

Anthropic is now blacklisted. I’m curious to see what happens next: will the other AI giants stand behind them and say: we won’t do this either? Or does someone like xAI raise their hand and say: Anthropic didn’t want that contract, we’ll take it? [Editor’s note: Hours after the interview, OpenAI announced its own deal with the Pentagon.]

Last night Sam Altman came out and said he stands behind Anthropic and has the same red lines. I admire him for having the courage to say that. As of the time we started this interview, Google had said nothing. If they just stay quiet, I think that’s incredibly embarrassing for them as a company, and many of their employees will feel the same way. We haven’t heard anything from xAI yet either. So it will be interesting to see. In short, there is a moment when everyone has to show their true colors.

Is there a version of this where the result is actually good?

Yes, and that’s why I’m actually strangely optimistic. There is such an obvious alternative here. If we’re just going to treat AI companies like all other companies – and drop the corporate amnesty – they should clearly do something like a clinical trial before releasing something this powerful, and show independent experts that they know how to check it. Then we will have a golden age with all the good things about AI, without the existential fear. That is not the path we are following now. But it could be.