The future of governing AI agents

How to build governance into autonomous security agents from the architecture up

Marcus Jeffes

July 8, 2026

Share on LinkedIn

Share on LinkedIn

Share on Facebook

Share on Facebook

Share by Email

Share by Email

Print this page

Print

The industry has moved fast on capabilities. Agents now triage alerts, investigate endpoints, create detection rules, enrich indicators, and are even capable of performing most actions we as security operators can perform. The architecture patterns are maturing, as are the models, but governance is not keeping pace. The discipline of proving your agents are doing the right thing continuously, under pressure, and at scale remains largely unsolved.

Most teams deploying agents in production today default to a tiered autonomy model where low-risk actions run autonomously, medium-risk get oversight, and high-risk require human approval. This tells you what an agent can do, but it tells you nothing about whether it's doing it well. A triage agent that classifies every alert as a false positive will never trigger a governance gate because it's operating within its approved tier.

Meanwhile, the regulations are not waiting. ISO 42001 , DORA, NIS2, and the EU AI Act are all converging on the expectation that you must prove your AI systems work as intended, that humans maintain meaningful oversight, and that you can explain how decisions are made. The assessment criteria specific to autonomous security agents don't exist yet, but the direction is clear. And the organizations building governance infrastructure now will be the ones ready when it arrives.

Reasoning as infrastructure

Right now, how an agent reasons about a security investigation is baked into whichever large language model (LLM) powers it. Beyond that, probabilistic models have a tendency to not come with consistent outputs each time. Swap the model, or even have the same incident occur and the reasoning patterns change. The same alert with the same context can and produce different escalation decisions. A governance framework built around a given model’s tendencies can become less reliable when the model changes, or the model deviates due to weights outside of our control.

Reasoning is the core determinant of agent behavior. It is what decides whether an alert gets escalated or closed, whether lateral movement gets investigated, or whether a host gets isolated. When that reasoning is baked into the model, you rely on the model's general intelligence to compensate for the absence of defined methodology. We need to extract the reasoning into an explicit layer for investigation workflows. This in itself has layers of applicability. This might look like a pre-defined, an instruction set or a knowledge base. We might invoke a workflow at runtime with baked in escalation logic. We might change our decision based on evidence evaluation criteria and/or hypothesis generation patterns. When we define reasoning at multiple layers, The model's role then reduces to execution of tasks. A well-defined methodology executed by a smaller, cheaper model will outperform a frontier model guessing at methodology for your specific environment because the reasoning is grounded in operational context rather than the model's training data. Allowing incidents to be handled faster and cheaper.

This matters in governance for three reasons.

1. Governance requires predictability.

Progressive trust assumes that the system being measured today is the same system that will operate tomorrow. If swapping a model changes reasoning patterns, the trust evidence accumulated under the old model doesn’t transfer; and you end up back in a maximum oversight state for every model change. The provider's concentration risk compounds this. If you need to switch providers due to pricing, deprecation, or regulatory designation, you lose both the model and the trust evidence.

1. Evaluation requires consistency.

If methodology is only consistent because of whichever model is running, you are unable to evaluate your own operations, as you will only ever be evaluating the model in question. Your ablation studies, benchmark suites, and quality metrics all become model-specific rather than system-specific, and every model change requires re-evaluation from scratch.

1. Compliance requires explainability.

When a regulator asks how does your agent decide to escalate an investigation?, the answer needs to be grounded in documented processes. If the answer comes down to ask the model provider, you can't demonstrate control. The regulator cares whether you can show them how decisions are made.

4-layer separation

The architecture that solves this separates into four layers.

1. Skills

Skills define what the agent can do across triage, enrichment, forensics, and detection engineering — each a composable unit of instructions, tools, and domain knowledge define the capabilities of the agent.

1. Reasoning

This defines how the agent thinks, which includes investigation methodology, escalation logic, evidence evaluation criteria, and hypothesis generation patterns. This layer is explicit, testable, and independent of which model executes it. When a triage decision follows a defined methodology, that methodology can be versioned, tested against benchmark suites, and updated through the same governance framework that gates every other consequential change. Because the methodology carries the complexity, the model doesn't have to. This is what makes it possible to run capable, governed agents on cost-effective models rather than defaulting to the most expensive frontier option for every task.

1. Models

While the LLM executes its own baked-in reasoning, the reasoning patterns are not the same for each model. Claude, Gemini, GPT, or open-source models all work either slightly or entirely differently, skewing results depending on how the model was trained and the guardrails put in place by the vendor.

1. Context

The depth of any investigation is bounded by the platform's ability to pull environmental context into the agentic solution. A clear record of every agent decision, what evidence grounded it, and whether it was correct is what makes continuous measurement and benchmarking possible.

This separation directly addresses these three problems:

The trust evidence attaches to the reasoning methodology and away from the model. When you swap models, trust also transfers because the methodology hasn’t changed. You verify the new model executes the same reasoning patterns correctly and can validate performance.

Benchmark suites test the reasoning methodology, but model changes require a targeted check, and evaluation must be a constant baked into our operation (i.e., does the new model follow the methodology to the same standard?).

When a regulator asks how the agent decides to escalate, you can point to the reasoning layer with versioned, documented, and testable outcomes. When they ask about model risk, you point to the separation layer where models are swappable infrastructure. When they ask about concentration risk, you demonstrate that reasoning patterns are portable across providers.

The idea of progressive trust is to start with maximum oversight, where every consequential action goes through a human. As evidence accumulates that the agent's recommendations are sound, oversight progressively relaxes. An agent that has been approved hundreds of times — with every recommendation justified in post-incident review — has earned a different quality of oversight than one deployed last week. This is only possible because the four-layer separation decouples trust from the model. The evidence attaches to the reasoning methodology, so it persists across model changes and accumulates over time rather than resetting with every update.

The inverse is equally important. If agent performance degrades due to model drift, data distribution changes, or novel attack patterns, we need metrics to surface it. When deterioration is observed, the response should be to tighten oversight by moving actions back to higher approval requirements, reduce auto-approve thresholds, and add evidence requirements. Trust earned over months can be revoked in seconds if the data warrants it.

But measuring trust requires the ability to observe not just what an agent did but also how it reasoned doing it.

Evaluation as metrics and evidence

Matt Kiely, principal security researcher at Huntress, argues the need for measuring progressive trust and how progressive trust requires evidence. That evidence comes from continuous evaluation across four dimensions.

1. Classification accuracy

Did the agent get the right answer? This is measured through precision (how many of its positive classifications were correct?) and recall (how many actual positives did it catch?). This measurement is necessary; however, this alone tells us nothing about reliability. A correct conclusion reached through hallucinated reasoning is not a reliable agent.

1. Planning quality

Did the agent consider multiple hypotheses? Given a suspicious login from two distant locations in four minutes, did it consider impossible travel, token theft, and VPN usage? Or, did it tunnel-vision on the first hypothesis?

1. Retrieval quality

Did the agent find the right evidence? If the knowledge base contains relevant prior incidents and the agent never retrieves them, is it operating on incomplete information?

1. Grounding quality

Is the agent’s reasoning supported by the evidence it found? Did it claim “this IP is associated with APT28 infrastructure” because a knowledge base entry said so or because it hallucinated that connection? An agent with a high unsupported claim rate is dangerous even when its conclusions happen to be correct.

These four dimensions compose into the evidence that progressive trust requires. Classification accuracy tells you whether to trust the outputs. Planning, retrieval, and grounding quality tell you whether to trust the process.

Ablation testing makes this concrete. Build a benchmark suite with known ground truth, run the agent, and capture the full trajectory — every tool call, every knowledge base query, and every reasoning step — then systematically vary the configuration:

Run with baseline only looking at the raw telemetry with no knowledge base, tools, or multistep reasoning. Can the model classify from signal alone?

Run without the knowledge base to isolate whether retrieval-augmented context actually improves outcomes

Run without investigation tools to determine whether tool use adds real value beyond what the model infers from telemetry and retrieved knowledge

Run the full agent with everything enabled as the comparison point

The difference between configurations reveals which components actually contribute to quality. If removing the knowledge base doesn’t change outcomes, either the knowledge base isn’t useful or the agent isn’t using it properly. These are the findings that feed back into system improvement.

Monitoring the monitors

When AI agents become operational in security, you will need them not only to be monitoring and responding to security incidents, but you also need to be able to monitor and respond to agent based decisions. This is where we need to shift and tie observability of our agents directly with our security capability. Traditionally, we as security professionals have been focused on actions occurred; now, we must switch to why they were performed:

What evidence did the agent consider?

What hypotheses did it explore and discard?

What reasoning led to action?

Did it hallucinate a connection between two unrelated events?

Did it skip a critical investigation step?

Did its confidence score reflect genuine signals or did it create one?

This semantic gap is telemetry and assessment of that telemetry. One customer that has built a system to assess agent behavior across their estate is Uber wit

[truncated for AI cost control]