Agent autonomy without guardrails is an SRE nightmare
João Freitas is GM and VP of engineering for AI and automation at PagerDuty
As the use of AI continues to evolve in large organizations, leaders are increasingly looking for the next development that will deliver big ROI. The latest wave of this ongoing trend is the adoption of AI agents. But as with any new technology, organizations must ensure they adopt AI agents responsibly so they can facilitate both speed and security.
More than half of organizations have already deployed AI agents to some extent, and more expect to follow suit in the next two years. But many early adopters are now reconsidering their approach. Four in 10 technology leaders regret not establishing a stronger governance foundation from the start, suggesting they adopted AI quickly but with margin to improve policies, regulations and best practices designed to ensure the responsible, ethical and legal development and use of AI.
As AI adoption accelerates, organizations must find the right balance between their exposure risk and implementing guardrails to ensure AI use is safe.
Where do AI agents create potential risks?
There are three key areas of focus for safer AI adoption.
The first is shadow AI, where employees use unauthorized AI tools without explicit permission, bypassing approved tools and processes. IT must create the necessary processes for experimentation and innovation to introduce more efficient ways to work with AI. While shadow AI has been around as long as AI tools themselves, the autonomy of AI agents makes it easier for unapproved tools to operate outside the reach of IT, which can introduce new security risks.
Second, organizations must close gaps in AI ownership and accountability to prepare for incidents or process failures. The power of AI agents lies in their autonomy. However, when agents act in unexpected ways, teams need to be able to determine who is responsible for addressing any issues.
The third risk occurs when there is a lack of explainability of the actions taken by AI agents. AI agents are goal-oriented, but how they achieve their goals can be unclear. AI agents must have explainable logic underlying their actions so that engineers can track and, if necessary, reverse actions that could cause problems with existing systems.
While none of these risks should delay adoption, they will help organizations better ensure their security.
The three guidelines for responsible adoption of AI agents
Once organizations have identified the risks that AI agents can pose, they should implement guidelines and guardrails to ensure safe use. By following these three steps, organizations can minimize these risks.
1: Make human supervision the standard
AI agency continues to develop at a rapid pace. However, we still need human oversight when AI agents are given the capacity to act, make decisions, and pursue a goal that can impact important systems. A human should be aware by default, especially in mission-critical use cases and systems. The teams using AI need to understand what actions they can take and where they may need to intervene. Start conservatively and over time increase the degree of agency given to AI agents.
Together, operations teams, engineers, and security professionals must understand the role they play in overseeing the workflows of AI agents. Each agent should be assigned a specific human owner for clearly defined supervision and responsibility. Organizations should also allow any human to flag or override an AI agent’s behavior when an action has a negative outcome.
When considering tasks for AI agents, organizations need to understand that while traditional automation is good at handling repetitive, rules-based processes with structured data input, AI agents can handle much more complex tasks and adapt to new information in a more autonomous way. This makes them an attractive solution for all kinds of tasks. But as AI agents are deployed, organizations must determine what actions the agents can take, especially in the early stages of a project. Thus, teams working with AI agents must have approval paths for high-impact actions to ensure that agents’ scope does not extend beyond expected use cases, minimizing risks to the broader system.
2: Safe baking
The introduction of new tools should not expose a system to new security risks.
Organizations should consider agentic platforms that meet high security standards and are validated by enterprise-level certifications such as SOC2, FedRAMP or equivalent. Furthermore, AI agents should not be given free rein within an organization’s systems. At a minimum, an AI agent’s permissions and security scope should be aligned with the owner’s scope, and any tools added to the agent should not allow extended permissions. Restricting AI agents’ access to a system based on their role also ensures a smooth deployment. Keeping complete logs of every action taken by an AI agent can also help engineers understand what happened in the event of an incident and trace the problem.
3: Make results explainable
The use of AI in an organization should never be a black box. The reasoning behind each action should be illustrated so that any engineer trying to access it can understand the context the agent used for decision-making and access the traces that led to those actions.
IInputs and outputs for each action must be recorded and accessible. This will help organizations gain a clear overview of the logic underlying an AI agent’s actions, which can provide significant value if something goes wrong.
Security underlines the success of AI agents
AI agents offer organizations enormous opportunities to accelerate and improve their existing processes. However, if they do not prioritize security and strong governance, they could expose themselves to new risks.
As AI agents become more common, organizations need to ensure they have systems in place to measure how they perform and the ability to take action if they cause problems.
Read more of our guest writers. Or consider posting yourself! See our guidelines here.
