Satya Nadella has issued a shocking warning to companies using AI

Of all the debates about the potential downsides of AI, there is one concern that is causing the most hand-wringing among AI enthusiasts in Silicon Valley. They fear that the giant AI labs selling proprietary models are somehow acting like Trojans.

The concern is that as startups and enterprises use AI models from labs like OpenAI and Anthropic, the labs will increasingly have access to those companies’ most sensitive business information. The model makers can then use that knowledge for themselves and possibly become competitors of their own customers. Those issuing such warnings range from venture capital firms like Jason Calacanis to Palantir CEO Alex Karp.

Now, in a surprising way blog post published on Monday, Microsoft CEO Satya Nadella has joined this crowd. Nadella warns that AI users (the ‘buyers’ as he calls them) pay twice. They knowingly spend money using AI tokens, but they also unknowingly hand over valuable data in the process.

“You’re essentially paying for intelligence twice, once with money, and again with something even more valuable: the proprietary knowledge you have to reveal to make that intelligence useful. The better you want the model to perform, the more of that knowledge you have to feed!” he writes.

The most dangerous thing is that companies literally teach the models the nuances of their business, he argues.

“Models learn from ‘exhaustion,’ the prompts people write, the tools agents use, and especially the corrections people make when the model is wrong. Every correction is distilled into institutional knowledge,” he writes.

This is “the kind of knowledge that a competitor could never buy,” and yet companies transfer this knowledge.

Nadella argues that if AI companies can freely scour the internet to train their models, it’s only fair that companies be allowed to study — or “distill” — those models in return. ‘Distillation’ is the practice of using a model’s own results to learn how it works and to train a new, often cheaper model based on those insights. In February, Anthropic accused Chinese open source models of sending millions of prompts to Claude as a way to improve their own models, and urged the US government to address export controls.

Nadella’s point is that model makers can’t have it both ways. It is hypocritical for them to freely train on the world’s data while restricting others from doing the same with their models.

“While the great innovation that comes from model vendors having fair use rights to train models on public data is necessary, I find it ironic that the status quo is then to turn around and impose restrictive conditions on distillation,” Nadella writes.

Nadella is particularly concerned that modelers “reserve the right to learn from customer usage and interaction data.”

Nadella’s solution is something the CEO of a giant cloud provider would propose. He wants companies to “retain ownership” of their data, including prompts, feedback, etc. So he urges them to build their own “native learning environments” in the cloud (where their data is probably already stored anyway and, conveniently, Microsoft’s cloud could mean Azure). He also wants companies to build in so-called “orchestration layers” – essentially a way to easily switch between AI models from different providers, rather than being locked into one model. Tools like AI “gateways” that allow companies to do just this have become increasingly popular.

While Nadella never uses the words “open source” as a method of maintaining ownership, this is an obvious subtext. Yet there is still a subtext.

Large companies, many of which still have some of their own data centers in addition to using the cloud, are already moving to open source models that are installed on their own premises (“on-prem”, in industry jargon). Idit Levine, founder and CEO of Solo.io — which makes networking and security software that helps companies manage AI systems — says she sees exactly this shift happening among her own customers. After experimenting with proprietary modelers, they’re starting to wonder, “Can I take an open source model and run it on premises? It will do almost 90% of what the big one does. It will cost a lot less,” she tells TechCrunch. “They understand that and can control it.”

Solo.io’s technology was selected last year as the technology that powers the The Linux Foundation’s Agent Gateway Project. Her company counts companies such as T-Mobile, ADP and SAP as customers. She sees companies increasingly installing on-premise open source models and sees this as the next big wave in enterprise AI use.

She’s not alone. Vercel (best known as a website building and hosting platform, which recently added AI model switching tools) and OpenRouter (a company that helps developers route requests through different AI models) are both seeing increases in traffic to open source models. In fact, open models were responsible for 29% of all routed traffic via the Vercel gateway last month.

With the CEO of Microsoft, a company that has invested in both OpenAI and Anthropic, now openly urging companies to be wary of using proprietary models, we’re betting this trend will continue to grow. “By consuming intelligence, you create intelligence. And what you create should be yours,” Nadella writes.