Why Large Language Models Hallucinate

A large language model can produce an answer that is clear, specific, well structured, and entirely wrong. That combination is what makes hallucinations more consequential in professional settings than an obvious error: the risk is not only that the model is wrong, but that it can be wrong in a form that looks reasonable.

For internal audit, the implication is immediate. AI-generated text can be valuable for exploring a hypothesis, summarizing documents, or drafting work. But fluency does not automatically increase evidential value.

The central point is simple: large language models are not designed as databases of guaranteed facts; they are probabilistic language-generation systems. The right control objective is therefore not to assume hallucinations can be eliminated, but to design AI use on the premise that a plausible output may still be false.

A convincing answer is not the same as a true answer

The NIST AI Risk Management Framework: Generative Artificial Intelligence Profile uses the term confabulation for cases in which a generative AI system presents erroneous or false content with apparent confidence. The same phenomenon is commonly called a hallucination.

The distinction matters because an LLM does not, by default, consult an internal table of verified facts and then write an answer. It generates statistically plausible sequences of language from the available context and patterns learned during training. That mechanism can produce remarkably useful results, but optimizing for linguistic plausibility is not the same as verifying truth.

A model can therefore invent a citation, attribute a requirement to the wrong standard, merge two real facts incorrectly, or fill a knowledge gap with an explanation that sounds right. NIST also notes that confabulation risk is particularly important in open-ended responses, specialist domains, and consequential decision contexts.

Linguistic quality is a property of the response. Reliability is a property of the evidence supporting it.

Why hallucinations occur

The basic mechanism does not require advanced mathematics. An LLM receives context and estimates which pieces of language are most likely to come next. It repeats that process until it forms a complete response. The result may match reality, but the generation process does not itself prove that every statement is true.

Four conditions make the problem more likely or more dangerous:

Insufficient or ambiguous context. When information is missing, the model may complete the pattern rather than explicitly state that it does not know.
Outdated or incomplete knowledge. The model may not have the current version of a standard, policy, regulation, or recent event.
Weak external sources. Even search-enabled or RAG systems can retrieve incomplete, irrelevant, or stale material and then produce a flawed synthesis.
User overreliance. A polished response lowers cognitive friction and can reinforce automation bias: people accept the output because it appears competent.

Better prompting can reduce ambiguity, request explicit assumptions, and require citations. That improves the interaction, but it does not turn the model into a primary source. RAG can add governed documents and traceability, but it is not a truth guarantee either: retrieval can fail, and the model can still omit, misread, or overstate what it retrieved.

Controls should be proportional to the consequence of error

The useful question is not “Does this model hallucinate?” A sound control environment should not depend on a binary answer. The better question is: What happens if this output is wrong and someone relies on it?

COSO’s 2026 guidance on internal control over generative AI starts from this characteristic. GenAI is probabilistic rather than deterministic, so outputs should be treated as claims that require validation rather than facts accepted by default. The guidance highlights controls such as pre- and post-deployment validation, risk-based human review, performance testing, source traceability, and error monitoring.

AI use	Primary risk	Proportionate control	Evidence an auditor would expect
Brainstorming or internal drafting	Low-consequence error	User review before reuse	Clear usage guidance and assigned accountability
Summarizing policies or standards	Omission, false citation, or stale content	Approved sources, freshness controls, and citation verification	Source inventory, version records, review samples, and change logs
Analysis incorporated into workpapers	Unsupported conclusion	Independent corroboration and traceability to original evidence	Cross-references, supervisory review, and documented procedures
High-consequence decision or action	Scaled or automated error	Thresholds, human approval, pre-deployment testing, monitoring, and rollback or stop mechanisms	Test results, logs, approvals, exceptions, and incident records

The same logic is consistent with ISO/IEC 42001:2023, which establishes requirements for an AI management system that addresses AI risks and opportunities, performance evaluation, documented information, and continual improvement. The standard does not promise infallible AI; it requires organizations to establish processes to govern, assess, and improve AI use.

For internal audit, AI output should not become evidence by accident

This is where the Global Internal Audit Standards become directly relevant. Standard 14.2 requires internal auditors to analyze relevant, reliable, and sufficient information when developing potential engagement findings. If an AI-generated assertion enters a workpaper without a defensible source and validation trail, the issue is no longer merely technical. It is an audit-evidence quality problem.

The IIA’s AI Prompting for Internal Auditors makes that practical. AI-generated content used in workpapers, reports, or advisory deliverables should be independently verified against authoritative sources and treated as a draft starting point rather than a final product.

That changes the discipline of use. An auditor may ask an LLM to identify contradictions between documents, propose procedures, or synthesize interview notes. But professional accountability remains human: determining what constitutes evidence, assessing reliability, seeking corroboration, and documenting the basis for the conclusion.

The IIA Artificial Intelligence Auditing Framework reinforces the same logic by positioning transparency, traceability, governance, and control as central elements of AI assurance.

Five questions that make hallucination auditable

An audit team assessing an LLM-based use case can start with five questions:

What decisions depend on the output? The greater the consequence, the lower the tolerance for unvalidated output.
What authoritative source can confirm the claim? If no verifiable source exists, the content should be labeled as a hypothesis or analysis rather than fact.
How is traceability retained? Prompts, sources, versions, configurations, reviews, and changes may be necessary to reproduce or investigate the result.
What happens when the model is wrong? Detection, correction, escalation, and where appropriate rollback or stop mechanisms should exist.
Who remains accountable? Introducing AI should not dilute the responsibility of the process owner, reviewer, or auditor.

These questions move the discussion from “How intelligent is the model?” to “How reliable is the system of use around the model?” That is the more useful assurance question.

The objective is not zero hallucinations; it is controlled reliance

Waiting for an LLM that never produces a false statement is a weak control strategy. A stronger architecture combines clear use boundaries, governed sources, testing, traceability, risk-proportionate human review, and ongoing monitoring.

For low-impact uses, reasonable review may be enough. For audit, compliance, financial reporting, regulatory decisions, or automated actions, the bar should be higher. The organization should be able to show why it relied on a particular output and what controls prevented a plausible error from becoming a real decision.

Hallucination does not make LLMs useless. It simply forces a discipline that auditors already understand: a claim does not become reliable because it is well presented; it becomes reliable when sufficient evidence supports it.