Hallucinations: From AI Error to Business Risk

An AI assistant produces a polished answer, cites a policy, and recommends what to do next. Nothing looks obviously wrong. The problem emerges later: the policy says something different, the citation does not exist, or the fact belongs to another context. At that point, the error is no longer confined to the model. It has become a business decision, communication, or action.

That is why hallucinations matter to Internal Audit. The useful question is not simply “Can the model be wrong?” We already know it can. The better question is what happens inside the organization when a false answer is convincing enough to be used.

Moffatt v. Air Canada is a useful accountability example. A chatbot on the airline’s website gave inaccurate information about bereavement fares that conflicted with another Air Canada page. The tribunal found the airline responsible for the information delivered through the chatbot and concluded that it had not taken reasonable care to ensure accuracy. The decision does not establish that the chatbot was a large language model or that the failure was a hallucination in today’s technical sense. Its audit lesson is more durable: customers deal with the organization, not with its technology as a separate actor.

A hallucination is not just “a wrong answer”

The NIST Generative AI Profile uses the term confabulation for erroneous or false content that a generative AI system presents confidently. The concept also covers outputs that diverge from the prompt or contradict statements made earlier in the same context. “Hallucination” is the more common informal term.

That distinction matters because conventional system failures often provide a visible signal: a formula breaks, a field is empty, or a rule throws an exception. A hallucinated answer may be grammatically clean, logically structured, and supported by references that look credible but are fabricated. Presentation quality can exceed evidential quality.

This behavior follows partly from how generative models operate. A language model produces plausible token sequences from the context and statistical patterns it learned. It does not necessarily consult a source of truth before composing each sentence. Fluency and factual accuracy are therefore separate properties.

The opposite overreaction is also unhelpful. Hallucination risk does not mean every AI output is unreliable. Risk depends on the use case, available sources, application design, verification process, and—most importantly—the consequence of being wrong.

The control problem is not that AI can produce a false statement. It is that an organization may allow that statement to travel too far without verification proportionate to its impact.

Risk begins when the output enters the workflow

The same model behavior may be acceptable in brainstorming and unacceptable in a payment process, regulatory submission, or audit conclusion. The risk therefore does not sit only inside the model. It emerges from the combination of output, context of use, authority to act, and the ability to detect an error.

Context	How hallucination may appear	Business consequence	Audit question
Process	Invents a requirement, supplier status, or operating step.	Rework, incorrect payments, noncompliance, or disruption.	Which outputs can trigger action, and what validation occurs first?
Decision	Misstates facts, alternatives, or decision criteria.	Approvals, pricing, credit, or priorities based on false premises.	Who verifies critical facts, against what evidence?
Reporting	Generates unsupported figures, explanations, or references.	Incorrect management or external reporting.	Is generated information reconciled to authorized systems and sources?
Customer	Invents a policy, condition, commitment, or explanation.	Complaints, remediation, legal exposure, and loss of trust.	How is accuracy controlled, and how are uncertain answers escalated?
Assurance	Fabricates a citation, source, or interpretation of evidence.	Unsupported findings or conclusions.	Is AI supporting analysis, or becoming a de facto source of audit evidence?

This framing makes the issue auditable. “Reduce hallucinations” is too vague to be a useful control objective. “Prevent an unverified answer from changing a master record, reaching a customer, or supporting an audit conclusion” is much more concrete.

There is no single anti-hallucination control

The stronger response is layered. NIST AI 600-1 recommends, among other actions, measuring performance under conditions similar to deployment, reviewing and verifying sources and citations during testing and ongoing monitoring, and maintaining post-deployment processes for risks such as confabulation.

In practice, those ideas translate into controls that reinforce one another.

Start by classifying the use case by consequence. Not every task needs the same control intensity. A low-stakes internal draft may tolerate lightweight review. Contractual language, credit decisions, regulatory reporting, or audit communications require a much higher threshold.

Next, constrain what knowledge the system can use. Retrieval-augmented generation, or RAG, can improve grounding by retrieving approved policies, contracts, or procedures before generation. But RAG does not eliminate hallucination risk. Stale, incomplete, poorly retrieved, or out-of-context sources can produce a well-cited answer that is still wrong.

Then design real verification. Asking the same model to “check its answer” is not independent validation. For material facts, organizations may need traceability to an authoritative source, reconciliation to source systems, or review by a competent person.

Also limit what can happen after the answer. As AI systems gain the ability to send messages, update records, approve transactions, or invoke tools, post-hoc review becomes a weaker safeguard. Permission boundaries, business rules, approval gates, and safe fallback behavior reduce the consequence of a false output.

Finally, monitor actual failures. Complaints, corrections, user overrides, invalid citations, exceptions, and near misses are valuable control signals. ISO/IEC 42001:2023 reinforces this management-system logic through risk assessment, monitoring, and continual improvement. COSO’s 2026 guidance on internal control over GenAI similarly places generative AI risk within the broader objectives of operations, reporting, and compliance.

For Internal Audit, the question becomes: what evidence are we accepting?

Hallucinations create a particular issue when Internal Audit uses AI itself. A tool can summarize interviews, compare policies, draft a finding, or suggest possible root causes. Those capabilities may improve speed and coverage. They do not change the professional standard for evidence.

Under Standard 14.1 of the Global Internal Audit Standards, internal auditors must gather information that is relevant, reliable, and sufficient for analysis and evaluation. Reliability is strengthened when information is obtained directly or from an independent source, corroborated, or collected from a system with effective governance, risk management, and control processes. An AI-generated statement does not become reliable evidence because it is fluent, contains citations, or agrees with the auditor’s expectation.

A generalist internal auditor can test a great deal without becoming a model specialist:

select material use cases and trace what happens from prompt to final decision or action;
inspect authoritative sources, freshness controls, and permissions used by RAG-enabled systems;
review pre-deployment tests using representative questions, incomplete data, and edge cases;
sample accepted, corrected, and rejected outputs to determine whether human oversight detects consequential errors;
verify that material claims and citations can be reconstructed from real source evidence;
review metrics, complaints, incidents, overrides, and escalation mechanisms after deployment.

Specialist support becomes more important when the engagement requires comparative model evaluation, technical factuality metrics, statistically valid error-rate testing, or analysis of model internals. Internal Audit can frame the assurance question and assess specialist evidence without pretending to replace that expertise.

The useful objective is controlled reliance, not zero hallucinations

“Zero hallucinations” is rarely a realistic governance objective. Treating hallucinations as an unavoidable excuse is equally weak. The organization must decide where uncertainty is tolerable, where verification is mandatory, and where an AI output should simply not have enough authority to create a material consequence on its own.

That is the point at which hallucination stops being a technical curiosity and becomes a familiar audit problem: objectives, risks, controls, evidence, and accountability.

AI may generate the answer. The organization remains accountable for what happens next.