The internet is being rebuilt for machines

Cloud infrastructure has long been designed around people searching, clicking, scrolling, and streaming in a stable and predictable way. AI agents behave differently. They can unleash a huge amount of activity, spawning multiple sub-agents within seconds that search hundreds of databases, search documents, and call APIs, then disappear as quickly as they arrived.

With that premise, Amazon is redesigning a core part of its cloud infrastructure. Thursday AWS launched its next generation OpenSearch Serverlessa fully managed search and vector database (essentially a system for storing and retrieving information at scale) designed specifically for agentic workloads. AWS says the new system can immediately scale up when agents activate tasks and scale back down to zero when idle.

The launch reflects a growing realization within the technology industry: infrastructure originally designed for a human-powered internet doesn’t work well in a world increasingly populated by agents.

While AI agents still represent a relatively small portion of internet activity, machine-generated traffic is already significant and poised to grow. Cloudflare says bots were responsible for 31% of total HTTP traffic over the past six months. AI crawlers, search engines and assistants accounted for about a quarter of all bot requests during that period.

“Non-human traffic will exceed human traffic sometime in the first half of 2027,” he says Lai Yi Ohlsensenior product manager at Cloudflare, told TechCrunch.

At Google’s I/O developer conference last week, the company said users will be able to start delegating tasks to AI systems, such as researching purchases, booking travel, browsing the web and interacting with apps. But the buck doesn’t stop with consumer-facing AI agents. Companies are increasingly deploying agents, both internally and for their customers, creating new forms of machine-generated traffic behind the scenes.

As a result, cloud providers and infrastructure companies have been thinking about how to adapt systems built for people to a world of agents that continuously and autonomously retrieve information, invoke tools, and generate traffic from machine to machine.

That’s where AWS’s new OpenSearch Serverless comes into the picture.

“The timing is clear. Agents are moving from experiment to production and they are creating traffic patterns that the previous infrastructure was simply not designed for,” Tia White, general manager of Amazon OpenSearch Service, told TechCrunch. “They spike without warning, they lie dormant without notice, and enterprises need a search engine that can keep pace without paying for empty or idle computing power.”

The key technical change in this new generation is the decoupling of compute from storage, allowing compute to scale up in seconds to meet agent traffic spikes and scale down to zero so that customers pay $0 when agents are idle.

“Previously, even in our previous serverless version, you had to have at least one instance up and running because storage and compute were tied together,” White said. “You couldn’t just start automatically [compute] at the speed you needed, so you always had idle computing power reserved for your workload, whether you used it or not.”

Think of it as always paying for a parking space, even if you don’t use it. With AWS’s upgraded Serverless, it’s more like paying for parking.

At launch, OpenSearch Serverless will natively integrate with AI development platforms such as Vercel and Kiro, allowing developers to deploy production-ready search and vector agent backends without having to manage infrastructure.

The shift is happening in the cloud industry. Databricks and Snowflake are repositioning themselves as AI memory and retrieval systems for enterprise data. Microsoft has rolled out updates for Azure designed to handle AI agent bursts and share memory between agents. Cloudflare, in the same vein as Amazon, introduced last month infrastructure focused on giving agents persistent environments and instant scalability.

The more companies deploy AI agents, the greater the pressure will be to redesign infrastructure around machine-generated workloads, which in turn could make agents cheaper and easier to deploy at scale.