How to Audit AI Agents

An AI agent may review an invoice, query an ERP record, check the contract, email a supplier, and prepare a payment action. The control problem emerges when the system can decide which tool to call, which information to combine, and which action to take with limited human involvement.

That changes the audit object. In a conventional generative AI application, assurance may focus heavily on output quality, data protection, and appropriate use. For an agent, those questions remain necessary but insufficient. The critical unit of control is the action chain: who authorized the agent, what it can access, what it can do, when approval is required, what state it retains, what it records, and when it must stop.

Move the audit from outputs to actions

An agentic architecture typically combines instructions, a model, context, and external tools. Those tools may read files, query databases, send messages, modify records, run code, or initiate transactions. An inaccurate answer may still be intercepted before use; an incorrect action can alter the environment before a reviewer sees it.

The NIST Artificial Intelligence Risk Management Framework (AI RMF 1.0), which NIST is currently revising, offers a useful structure through Govern, Map, Measure, and Manage: establish accountability and boundaries, understand context and consequences, measure actual behavior, and manage incidents, changes, and continued deployment. NIST also launched its AI Agent Standards Initiative in 2026 to address autonomous agents and related security, identity, interoperability, and trust issues.

ISO/IEC 42001:2023 provides a complementary management-system lens. It is not agent-specific, but its requirements for roles, risk management, planned changes, operation, monitoring, and documented information, together with lifecycle, responsible-use, and event-logging controls, provide relevant criteria for the environment in which an agent operates.

The audit question is therefore not merely, “Does the agent work?” but “Can it act only within the authority the organization deliberately granted?”

Start with the agent's authority map

Before designing tests, auditors should reconstruct the authority map: which identity the agent uses, which tools it can invoke, what permissions those tools confer, on whose behalf it acts, and which decisions remain outside its authority.

NIST's concept-stage work on Software and AI Agent Identity and Authorization is useful as an emerging risk signal, not a mandatory control standard. It explores least privilege, proof of authority for specific actions, “on behalf of” delegation, binding agent activity to human authorization, and traceability. For auditors, broad access justified because “the agent might need it” is a warning sign. Shared credentials that make agent and human actions indistinguishable weaken accountability as well.

Eight control surfaces that should withstand testing

The matrix below turns principal agent risks into evidence and audit procedures. It should be tailored to the use case, autonomy level, data sensitivity, and consequences of an incorrect action.

Control surface	Primary risk	Expected evidence	Audit test
Tool access	The agent can use unapproved tools or data sources.	Tool inventory, architecture, connectors, ownership.	Compare runtime-accessible tools with the approved inventory and verify an unapproved tool is inaccessible.
Permissions and delegation	Excess privilege allows the agent to read, change, or execute more than needed.	Agent identity, roles, scopes, tokens, authorization matrix.	Test least privilege and attempt an action outside the authorized scope.
Action boundaries	A legitimate request expands into an unintended action.	Use-case specification, allowed/prohibited actions, business rules.	Create boundary cases and verify refusal, escalation, or renewed authorization.
Approval gates	A consequential action occurs without meaningful review.	Approval thresholds, matrix, records, reviewer competence requirements.	Reperform high-impact cases and confirm execution cannot complete before valid approval.
Memory and state	Prior context crosses users, alters later actions, or persists sensitive data too long.	Memory design, retention, isolation, deletion, reset rules.	Seed controlled state and test isolation, persistence, and deletion against design.
Logging and traceability	The organization cannot reconstruct what the agent did or who authorized it.	Action and tool logs, errors, approvals, identities, timestamps, outcomes.	Sample executions and reconstruct the end-to-end action sequence.
Exceptions and recovery	The agent continues after ambiguity, tool failure, or partial execution.	Exception rules, review queues, rollback, human takeover procedures.	Force foreseeable failures and verify escalation, rollback, or transfer of control.
Stopping conditions	The agent loops or continues after risk exceeds a limit.	Time, cost, action limits; alerts; deactivation procedure.	Trigger a stop condition and verify that new actions cease and intervention is logged.

Two nuances matter. Agent memory is not a standalone ISO/IEC 42001 control category; audit it as governed state and data: source, persistence, isolation, retention, and influence on future actions. Likewise, a “human approval” box on an architecture diagram does not demonstrate effective oversight. Approval must occur before the consequential action, provide enough information for a competent decision, and give the reviewer real authority to reject, modify, or stop it.

ISO/IEC 42001 reinforces this logic through controls addressing requirements, verification and validation, deployment, operation and monitoring, technical documentation, and event logging. Auditors should translate those concepts into the actual agent architecture rather than accept a generic AI policy as sufficient evidence.

Test scenarios, not just settings

Configuration demonstrates design; it does not prove that controls operate during a real run. An advanced audit should combine at least three scenario families: the normal path, to observe tool calls, approvals, and results; boundary and negative cases, to request out-of-scope actions or exceed thresholds; and untrusted content, to test whether external instructions can distort the legitimate task or granted authority.

The third category matters especially when agents consume emails, files, or websites. NIST's technical work on AI agent hijacking shows how indirect prompt injection can induce unintended actions. Internal Audit can design controlled cases, while deep adversarial testing may require cybersecurity or AI specialists.

State, recovery, and stopping should also be exercised. Can contaminated memory influence the next task? Can a partial action be reversed? Can an operator take over without losing traceability? The NIST AI RMF Playbook connects post-deployment monitoring with override, incident response, recovery, change management, and decommissioning. For agents, those practices should become observable limits on time, steps, spend, action type, data exposure, or other use-case-specific thresholds.

Evidence should reconstruct what happened

A strong agent audit file is not a collection of chatbot screenshots. It should reconstruct a complete run by connecting the initiating request to agent identity, relevant context, tool calls, authorization decisions, state changes, human approvals, exceptions, and outcome.

ISO/IEC 42001 includes controls for determining where event logging is needed and, at minimum, enabling event records while the AI system is in use. Sampling should therefore focus on complete executions, including successful runs, refusals, exceptions, high-impact actions, and incidents. Logs should be complete and synchronized enough for another competent person to follow the sequence without relying on a developer's oral explanation.

Mature autonomy is bounded and observable autonomy

Agent maturity is not measured by how many decisions are removed from human involvement. It is measured by how clearly the organization defines authority, constrains action, observes behavior, and can intervene.

Internal auditors do not need to become agent developers to test distinct identity, least privilege, meaningful approvals, governed memory, traceability, and stopping conditions. Models and protocols will change; an audit approach centered on authority, action, evidence, and intervention remains durable because it examines not only what the agent says, but what the organization has allowed it to do.