How AI guardrails are impeding the work of offensive cybersecurity researchers

For months, AI giants have devised special controlled programs and strict guardrails to limit the use of their models by malicious hackers. But these restrictions are now hampering the work of legitimate network defenders, as well as that of offensive cybersecurity researchers.

In June, the US government imposed export control restrictions on Anthropic’s much-hyped AI models Mythos and Fable. The move was at least partially prompted by a report that claimed it was possible to bypass the models’ guardrails, which were intended to prevent users from using them to build and execute malicious cyberattacks.

Regardless of whether the incident was actually motivated by the fear of a jailbreak, the fact is that Anthropic has repeatedly marketed Mythos as some kind of doomsday cyber machine that can only be given to carefully vetted users, and even then with strict guardrails. (Export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 resumed general access on July 1; Mythos 5 has been reintroduced only to vetted US organizations as part of the government’s review process.)

That kind of gatekeeping is not unique to Mythos. Both Anthropic, with its other models, and OpenAI offer programs for cybersecurity researchers to apply to be vetted and – if approved – gain access to models with fewer cybersecurity restrictions: OpenAIs Trusted access for cyber and Anthropic Cyber ​​verification program.

These guardrails have been widely criticized, especially by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.

During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, said: said that “I don’t really like these random big companies making arbitrary decisions about what’s safe in terms of security and what’s not.”

Dowd spent decades Finding and selling “zero days”. – previously unknown software bugs and the exploits that exploit them – to Western governments, instead of reporting them to the software makers so that they are patched. Governments pay a premium for vulnerabilities precisely because they remain open, which is useful for intelligence operations.

Dowd admitted that his work may make him biased, but he’s not alone. Several people who work in offensive cybersecurity—proactively examining systems for weaknesses—described to TechCrunch how they use AI tools and handle their guardrails.

Chris Anley, the chief scientist at security consulting giant NCC Group, said asking an AI model to try to exploit a bug is an important step in confirming that it is a real vulnerability worth fixing. But if a guardrail causes the model to avoid answering the question outright, the guardrail hurts defenders, he said.

“This is where the whole offensive versus defensive and guardrail part comes in, because ‘fix this code’ as a prompt is both an essential defense mechanism, but also a roadmap for finding critical vulnerabilities in the code base,” Anley said. “So at the same time, the same instrument is both an offensive and a defensive instrument, and the two cannot really be separated.”

It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s certainly a tool, but it’s also irreducibly a weapon.”

When he and his colleagues encounter such an obstacle, they sometimes fall back on open-source AI models that have no guardrails at all.

Paolo Stagno, the chief technology officer at CrowdFense, a well-known company that develops, acquires and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies are “essentially treating customers like children who need to be babysat” with their vetted programs and guardrails.

Stagno said he and his colleagues do use boundary models, but only for reverse engineering. They avoid using AI to help find vulnerabilities or build exploits, he said, because feeding that work into a cloud-based model risks sensitive vulnerability data leaking or being included in future training sessions. For that step, he says, they use open source models that run locally because they don’t rely on sharing data outside the model.

Giuseppe Cali, a security researcher who finds zero-days and develops exploits, said guardrails don’t hinder his work. That’s because he doesn’t use AI for attacking work; instead, he uses it for initial reverse engineering, to understand the code he analyzes and to build supporting tools. Therefore, he said, AI tools can speed up the process and allow him to focus on discovering vulnerabilities.

“I still want to own the actual discovery and weaponization of bugs and that wouldn’t change if all the guardrails were lifted tomorrow,” Cali said. “I’m jealous of my bugs, and I like this game too much to let models play it for me.”

A researcher at a smartphone component maker, who spoke on condition of anonymity because he is not authorized to talk to the press, said his employer is not part of Anthropic’s CVP program and therefore the tools are hardly useful for finding vulnerabilities because the guardrails are too strict.

“If wind comes up, we do something with the protection, it just stops and is no longer usable,” the person said.

Chris Thompson – CEO of cybersecurity company RemoteThreat and founder of Offensive AI Con, an offensive security and AI-focused event – ​​said that in his experience using cutting-edge AI models, the guardrails can be inconsistent and operate differently every day. That’s true even within the looser boundaries of Anthropic and OpenAI’s vetted programs.

“I think the practical impact is that you spend a lot of time negotiating the model rather than working on the core security program,” Thompson said. “Instead of analyzing a vulnerability and reasoning about its exploitability, you try to figure out why you get inconsistent results or why models over-sanitize the output.”

As a result, researchers are relying on or being pushed toward Chinese open-source models such as GLM – free downloadable models that can be run locally without control or usage restrictions – Thompson said.

“You have responsible researchers being pushed away from US-controlled systems to foreign-owned systems,” he said. “I think it would be more harmful than good to install these guardrails.”

Rather than tighten restrictions further, Thompson called on AI frontier labs to open up their programs, provide responsible access and also hold accountable those who misuse their tools. Otherwise, he argued, defenders will lose the AI ​​race.

“There’s a big storm coming. There’s a big wave of attacks that will happen with speed and scale like never before,” Thompson said. “But the same security consultancies and legitimate researchers trying to make a difference are currently being suppressed.”