Skip to main content

AI Governance

Governing what AI agents do

AI governance for agents is the practice of deciding, enforcing, and evidencing what an AI agent is permitted to do in your enterprise systems. It differs from model governance, which covers how a model was built and how it performs, and from content moderation, which covers what a model says. Once an agent can call tools and change records, the governing question becomes whether a given action is authorized — and that has to be answered before the action executes, not detected afterwards.

Why agents need their own governance layer

Enterprise controls were built for people making decisions. IT service management, SOX Segregation of Duties, and access review cycles all assume a human actor whose behaviour sits well inside their permissions, held there by training, judgement, and the friction of having to decide.

None of that applies to an agent. An agent connected to a system can call every operation that system exposes, immediately, and it will if its reasoning concludes it should. There is no gap between access and behaviour for organisational friction to fill, which is why an agent's exposure equals its reach rather than its role.

The three tracks below cover how that gets addressed: defining the boundary, enforcing it in the path of the action, and proving afterwards that it held.

OWASP LLM06:2025 · AIUC-1 D003

The risk has a name. So does the control.

The risk

Excessive Agency

OWASP catalogues excessive agency as LLM06:2025: an agent given the ability to call tools can take damaging actions on the back of output that is unexpected, ambiguous, or deliberately manipulated. It traces the risk to three causes: excessive functionality, excessive permissions, and excessive autonomy. The mitigations it names are to limit an agent to the minimum tools it needs, scope every permission to least privilege, and put a human in front of high impact actions.

Read LLM06:2025 on OWASP

The control

Restrict unsafe tool calls

AIUC-1 turns that guidance into something an auditor can test. Control D003 is mandatory and preventative, requiring safeguards that prevent tool calls in AI systems from executing unauthorized actions, accessing restricted information, or making decisions beyond their intended scope. The evidence it expects is specific: per agent tool authorization and parameter validation, rate limits, logging of every tool call, and human approval workflows for high risk operations.

Read D003 on AIUC-1

Both point at the same place: the moment a tool call is about to execute. LangGuard authorizes that call deterministically before it runs, against the action surface mapped for that agent at design time. Anything crossing a Segregation of Duties boundary or a policy threshold routes to a named approver, and every decision lands in the audit ledger as evidence you can hand to a reviewer. See how MCP Context Authorization enforces it.

Compliance

Regulatory alignment

Where LangGuard's controls map to the frameworks enterprises are audited against. This is a control mapping, not a certification claim: the obligations belong to you as the deployer, and these are the capabilities that produce the evidence for them.

LangGuard capabilities mapped to regulatory and control frameworks
Framework Obligation LangGuard capability
EU AI Act — Article 14, Human oversight High-risk systems must be designed so a person can oversee operation and intervene or interrupt it. Arbiter holds any action that crosses a policy threshold and routes it to a named approver before it executes, rather than raising an alert after.
EU AI Act — Article 12, Record-keeping Automatic recording of events over the lifetime of the system. Every decision — allow, block, escalate — is written to a tamper-evident action ledger with the agent identity, the policy that fired and the approver.
EU AI Act — Article 26, Deployer obligations Deployers must assign human oversight to competent, authorized people and retain the generated logs. Approvers are named per policy scope, so each approval records who held the authority; the ledger is the retained log.
NIST AI RMF — GOVERN, MAP Maintain an inventory of AI systems and document their risks in the context of use. SCOPE-MCP maps and compliance-classifies each agent's complete action surface at design time, backed by a live registry of agents and the tools they reach.
NIST AI RMF — MEASURE, MANAGE Monitor identified risks on an ongoing basis and act on them. Runtime tracing and anomaly detection for observed behavior, plus deterministic pre-execution enforcement for the risks you have already decided are unacceptable.
ISO/IEC 42001 — AI management system Documented controls, objectives and operational evidence across the AI lifecycle. Design-time classification and runtime decision records give per-agent evidence for both halves of the lifecycle, generated as a by-product of enforcement.
SOX — Segregation of Duties Whoever initiates a transaction must not be the one who approves it. SoD boundaries are evaluated deterministically on every agent action; a crossing is blocked or escalated to a named approver, never reconciled after the fact.
SOC 2 — CC6 access, CC7 monitoring Restrict access to what is authorized, and monitor system operation for deviations. Least-privilege scoping per agent enforced at the tool call, and a complete record of every authorization decision including the denials.

For the AI-specific standards — OWASP LLM06:2025 Excessive Agency and AIUC-1 D003 — see the control mapping above. For the full obligation-by-obligation breakdown, including where the deployer's own duty begins, see compliance-driven AI governance.

Common questions

What is the difference between AI governance and model governance?

They govern different objects and you need both. Model governance covers the model — how it was trained, evaluated, versioned, and whether it performs acceptably — and is largely a data science and MLOps discipline. Agent governance covers what a system built on that model is permitted to do in your enterprise: which tools it may call, which operations it may perform, and which of those need a human to approve them first. A perfectly governed model can be wired to a tool it should never have been given and take an action nobody authorized, because nothing about model provenance constrains an action surface.

Who owns AI agent governance in an enterprise?

In practice it lands across four groups, which is why it is often nobody's until an incident. Security and GRC own the control requirements and answer to auditors. IT owns the registry of what is running and who owns each agent. The teams building agents own the scoping decisions, because they choose the integrations that create the action surface. Internal audit and the risk function consume the evidence. The workable pattern is one shared policy model that all four read from, rather than each maintaining its own view — which is the thing that breaks when governance is assembled from separate design-time and runtime tools.

Find out what your agents can do — before your auditor does.

LangGuard maps your complete agent action surface in minutes. Free for the first five managed agents, with every Scope included from day one.