Rogue agents and shadow AI: Why VCs are betting big on AI security

What happens when an AI agent decides that the best way to complete a task is to blackmail you?

That’s not hypothetical. According to Barmak Meftah, partner at cybersecurity VC firm Ballistic Ventures, this recently happened to an employee of a company working with an AI agent. The employee tried to suppress what the agent wanted to do and what he was trained to do, and responded by scanning the user’s inbox, finding some inappropriate emails, and threatening to blackmail the user by forwarding the emails to the board of directors.

“In the agent’s mind, he’s doing the right thing,” Meftah told TechCrunch on last week’s episode of Equity. “It’s trying to protect the end user and the enterprise.”

Meftah’s example is reminiscent of Nick Bostrom’s AI paperclip problem. That thought experiment illustrates the potential existential risk of a super-intelligent AI that single-mindedly pursues a seemingly innocent goal – making paperclips – to the exclusion of all human values. In the case of this corporate AI agent, the lack of context around why the employee was trying to ignore his goals led him to create a subgoal that removed the obstacle (via blackmail) so he could achieve his primary goal. That combined with the non-deterministic character of AI agents means that “things can go rogue,” Meftah said.

Misaligned agents are just one layer of the AI ​​security challenge that Ballistic’s portfolio company Witness AI is trying to solve. Witness AI says it monitors AI use within enterprises and can detect when employees are using unapproved tools, block attacks and ensure compliance.

Witness AI raising $58 million this week thanks to over 500% growth in ARR and scaling its workforce by a factor of five over the past year as companies look to understand the use of shadow AI and scale AI safely. As part of Witness AI’s fundraising, the company has announced new agentic AI security measures.

“People are building these AI agents that take over the authorizations and capabilities of the people who manage them, and you want to make sure that these agents aren’t acting deceitfully, aren’t deleting files, aren’t doing anything wrong,” Rick Caccia, co-founder and CEO of Witness AI, told TechCrunch on Equity.

Meftah sees the use of agents growing “exponentially” across the enterprise. Complementing that increase – and the machine speed level of AI-powered attacks – analyst predicts Lisa Warren that AI security software will be an $800 billion to $1.2 trillion market by 2031.

“I think runtime observability and runtime frameworks for security and risk are going to be absolutely essential,” Meftah said.

On how such startups plan to compete with big players like AWS, Googling, Salesforce and others who have built AI management tools into their platforms, Meftah said: “AI safety and agentic safety are so big that there is room for many approaches.

Many companies “want a standalone platform, end-to-end, to essentially provide that observation and governance around AI and agents,” he said.

Caccia noted that Witness AI sits at the infrastructure layer and monitors interactions between users and AI models, rather than building safety features into the models themselves. And that was intentional.

“We purposely picked a part of the problem that OpenAI couldn’t easily get you into,” he said. “So it means that we end up competing more with the old security companies than with the model guys. So the question is: how can you beat the competition them?”

For his part, Caccia doesn’t want Witness AI to be one of the startups just acquired. He wants his company to grow and become a leading independent provider.

“CrowdStrike did it at the end point [protection]. Splunk did it in SIEM. Okta did it out of identity,” he said. “Someone comes along and stands next to the big boys… and we built Witness from day one to do that.