Testing Generative AI Outputs in Internal Audit

A generative AI application answers nine test questions correctly and fails the tenth. Does the control work? The answer depends less on the percentage than on which question failed, what risk it represented, and what behavior was expected before the test was run.

That is the central challenge of auditing generative AI. A model is not a deterministic rule that produces the same result every time. It may vary between runs, respond confidently when it should express uncertainty, refuse a legitimate request, or comply with a request it should block. Testing outputs therefore is not about collecting interesting responses. It is about building evidence of system behavior in the intended context of use.

The practical thesis is straightforward: GenAI output testing becomes defensible audit work when it starts with predefined criteria, covers risk-based scenarios, repeats enough interactions to observe variability, and preserves an evidence trail that another auditor can follow.

Test expected behavior, not a perfect sentence

Before drafting test prompts, the auditor needs a clear definition of what “working correctly” means. The Global Internal Audit Standards require internal auditors to identify relevant evaluation criteria and develop a work program specifying the tasks, methodologies, and tools needed to achieve the engagement objectives. For GenAI, that means translating business objectives and controls into observable behaviors.

For a policy assistant, for example, the criterion should not be “the answer looks good.” It might be: identify the applicable policy correctly, avoid inventing requirements, cite an authorized source, state when evidence is insufficient, and avoid disclosing restricted content to an unauthorized user.

This is consistent with the NIST AI RMF: Generative AI Profile, which recommends assessing output accuracy, quality, reliability, and authenticity against known ground truth and using complementary evaluation methods. NIST also stresses that testing should reflect the actual context of use. A generic benchmark or one impressive demonstration does not establish reliability in the process being audited.

The implication is important: the expected output does not need to be an exact string. In a probabilistic system, two differently worded responses may both be acceptable. What should remain within defined limits is the meaning, evidence basis, boundaries, and required safeguards.

Build a test portfolio around risk

A strong test set combines ordinary business cases with scenarios that force the system toward its boundaries. The purpose is not to “break the model” for entertainment. It is to determine whether controls still work when the likelihood or consequence of failure increases.

Test family	Example	Expected behavior	Useful evidence
Normal use	Common in-scope request	Correct, relevant, traceable response	Prompt, output, source, evaluation
Ambiguity or missing information	Required fact is absent	Requests clarification or states limitation	Response and escalation criterion
Edge case	Unusual format or rare combination	Preserves business rules or routes for review	Output, exception, human review
Grounding and citations	Answer exists in an approved repository	Uses the right source and does not fabricate citations	Retrieved source, citation, output
Prohibited request	Out-of-policy or sensitive request	Appropriate and consistent refusal	Refusal rationale and event record
Manipulation	Instruction attempts to override rules or extract restricted data	Maintains safeguards	Adversarial prompt, output, alert

ISO/IEC 42001:2023 reinforces this discipline through the AI system lifecycle. Organizations are expected to define and document verification and validation measures and criteria; its implementation guidance points to testing methods, test-data selection, and release criteria. For internal audit, this turns “Was the AI validated?” into testable questions: what was evaluated, against what criteria, with which data, who approved it, and what happens when the criteria are not met?

Sample for risk and repeat for variability

Traditional audit sampling addresses the cost of examining an entire population. GenAI adds another complication: the same interaction can produce different outputs. The population is therefore not only the set of query types. It also includes possible executions under defined configurations and contexts.

Sampling should start with risk. Scenarios with financial, regulatory, privacy, safety, or people-related consequences warrant more coverage and more repetitions than low-impact queries. Within each stratum, auditors should vary wording, include edge cases, and use conditions that reflect actual workflows. Where production logs provide a large population of outputs, analytics can complement manual testing by identifying anomalies, shifts in behavior, or concentrations of failure.

The Standards provide useful discipline. Standard 13.6 explains that, when sampling is used, the work program should document the sampling methodology, population, sample size, and whether results can be projected. Standard 14.1 asks auditors to consider testing a complete population or a representative sample and requires evidence to be relevant, reliable, and sufficient. There is no universal number of prompts that “tests an LLM.” The auditor should be able to explain why the coverage is sufficient for the conclusion being drawn.

For consistency, repeat critical cases and use semantically equivalent prompts. Do not demand word-for-word identity. Look for material variation in facts, calculations, sources, risk classifications, recommended actions, exception handling, or escalation decisions. COSO's 2026 Achieving Effective Internal Control Over Generative AI reflects this probabilistic reality and describes a range of control approaches, including human review, test populations, independent model or benchmark comparisons, analytics-based monitoring, and independent validation depending on risk.

Refusals are outputs too

A safe system must not only answer well; it must also refuse correctly. NIST recommends that acceptable-use policies define the types of queries a generative application should refuse. That creates two opposite risks for internal audit to test.

The first is under-refusal: the system answers a prohibited request, reveals information, ignores a restriction, or takes an action outside its authority. The second is over-refusal: the system blocks legitimate requests so often that users adopt workarounds, unauthorized tools, or parallel manual processes.

A refusal test should determine at least three things: whether the prohibited category is recognized, whether the response avoids exposing the information the control is intended to protect, and whether there is a safe path forward when the request is legitimate but incomplete. In many cases, the right behavior is not simply “I cannot help.” It may be to ask for missing context, escalate to a human, or provide only the portion of the response the user is authorized to receive.

The IIA Artificial Intelligence Auditing Framework places this kind of testing inside a broader assessment of governance, risk, data, algorithms, controls, and accountability. Internal audit should avoid allowing an isolated technical test to replace the assurance question: are the controls producing acceptable behavior for the intended use and risk profile?

Document a test another auditor can reproduce

GenAI test documentation needs more than a screenshot. Standard 14.6 requires engagement information, analyses, evaluations, and supporting evidence to be documented so that an informed and competent person can repeat the work and derive the same engagement results. For a generative system, “same results” is best understood as a reproducible audit conclusion, not necessarily identical text.

A strong workpaper should retain, where available: test-case ID and objective; related risk and control; prompt and supplied context; model or service version; relevant settings; system-prompt or rule-set version; retrieval sources or corpus version; date and time; number of repetitions; outputs obtained; acceptance criteria; evaluation result for each run; reviewer; exceptions; and evidence of retesting after remediation.

It also helps to separate three layers: what the system produced, how it was evaluated, and what the auditor concluded. This matters even more when Internal Audit uses GenAI to help classify or review the outputs under test. The IIA's AI Framework Support Tools reinforce the need to retain prompt information, document how outputs were validated and reviewed, and distinguish AI-assisted analysis from the auditor's professional conclusion.

Competence boundaries still matter. An AI-literate internal auditor can design scenarios, execute behavioral tests, inspect evidence, and assess controls. Advanced statistical validation, fairness measurement, deep adversarial red teaming, architecture security testing, or model-level validation may require specialists. Bringing in specialist support does not transfer the auditor's accountability; it improves the quality of evidence where the assurance question exceeds the team's technical capability.

Rigorous GenAI testing does not require Internal Audit to become a model laboratory. It requires applying familiar assurance discipline to a less deterministic technology: define criteria, test realistic conditions, obtain sufficient evidence, and document why that evidence supports the conclusion. When that chain is clear, the auditor can assess not merely whether the AI “gave a good answer” once, but whether the organization can reasonably rely on its behavior within defined boundaries.