Prompting, RAG and Agents: What Changes

A team starts with a generative AI assistant that summarizes policies. Later, the same-looking interface can answer questions using internal procedures. Later still, it can open applications, collect information, create records, or trigger workflow steps.

To the user, all three may look like a chatbot.

To an auditor, they are different systems.

The simplest distinction is this:

Prompting shapes what the model generates. RAG adds retrieval. Agents add the ability to take or coordinate actions.

The underlying language model may even be the same. What changes is the architecture around it: the information the system can access, the authority it can exercise, the dependencies it introduces, and the evidence required to assess whether it is controlled.

That is why prompting, RAG, and agents should not be treated as three labels for the same AI use case.

Generate, retrieve, act

A useful way to compare the patterns is to ask where the added capability comes from.

This is not a maturity ladder.

An agent is not automatically better than a RAG solution, and RAG is not automatically better than prompting. The right architecture depends on the business objective, the information required, the consequence of error, and the degree of authority the system genuinely needs.

A simple, well-controlled prompt-based workflow can be more appropriate than an autonomous solution with weak governance.

Prompting: the model generates, the human still drives the workflow

A prompt is the instruction or input a person gives to an AI system. It can include a task, context, examples, constraints, reference material, or a required output format.

For an internal auditor, prompting may be used to:

• summarize a policy;
• generate interview questions;
• structure a risk assessment;
• compare control descriptions;
• identify possible gaps in a work program;
• draft or refine an audit finding.

The value comes from flexibility. A user can change the task without rebuilding the system, and structured prompting can materially improve the usefulness of the output.

But a better prompt does not make the output authoritative.

The IIA's 2026 AI Prompting for Internal Auditors guidance emphasizes that AI-generated content can be incorrect, outdated, or fabricated and should be independently verified against reliable sources before being incorporated into workpapers, reports, or advisory deliverables. It also stresses the need for approved tools, data-handling rules, review, and documentation.

For a prompt-based use case, the control boundary therefore remains relatively close to the user.

Typical controls include:

• approved AI tools and acceptable-use rules;
• restrictions on confidential or regulated information;
• prompt guidance for material or recurring tasks;
• independent verification of significant outputs;
• clear accountability for the final conclusion;
• documentation where AI materially contributes to audit work.

Useful audit evidence may include the prompt or instruction, the information supplied to the model, the generated output, evidence of review, and the final human conclusion.

The central audit question is:

Was the model given appropriate information, and was the resulting output verified before anyone relied on it?

RAG: the system gains a retrieval layer

Retrieval-augmented generation, or RAG, changes the architecture.

Instead of relying only on the model's existing training and whatever the user manually provides, the system searches a defined body of information, retrieves material considered relevant, and places that material into the model's context before an answer is generated.

For Internal Audit, a RAG knowledge base might contain:

• audit methodology;
• corporate policies;
• control libraries;
• prior audit reports;
• regulatory guidance;
• procedures and standards;
• risk taxonomies.

This can make the system substantially more useful. An auditor could ask, "What does our methodology require before closing a finding?" or "Which prior audits identified third-party access issues?" and receive an answer grounded in internal content.

RAG can improve grounding. It does not guarantee correctness.

The system can still retrieve an incomplete source, miss the most relevant document, use an outdated version, expose information the user should not see, or generate a conclusion that goes beyond the retrieved evidence. A citation may look reassuring while pointing to a source that only partially supports the claim.

The control question therefore changes.

With prompting, the auditor mainly asks whether the information supplied to the model was appropriate and whether the response was reviewed.

With RAG, the auditor must also ask:

Can we trust what the system retrieved?

That introduces controls over:

• approved source repositories;
• ownership and quality of source information;
• document versioning and freshness;
• ingestion and indexing processes;
• access permissions and segregation of restricted content;
• retrieval quality and evaluation;
• traceability between generated statements and supporting sources;
• testing after changes to the knowledge base or retrieval configuration.

Audit evidence expands accordingly. It may include repository inventories, source ownership, access matrices, ingestion logs, document versions, retrieval logs, evaluation results, configuration changes, and samples showing whether the cited material actually supports the generated answer.

A useful rule is:

RAG does not remove model risk. It adds retrieval risk.

Agents: the system moves from answering to acting

The term AI agent is used broadly, so auditors should avoid relying on the label alone.

A practical definition is a system that can pursue an objective by selecting or sequencing steps and interacting with tools, systems, or data, often while maintaining some form of state or memory.

The important distinction is operational reach.

A copilot may recommend what a user should do next. An agent may be allowed to do it.

An audit-related agent could, for example:

1. receive a set of controls to review;
2. locate evidence in approved repositories;
3. identify missing documentation;
4. request additional evidence;
5. update an audit-management workflow;
6. prepare a draft workpaper for review.

At this point, the system is no longer only generating information. It is participating in a process.

That changes the consequence of error.

A poor chatbot response usually requires a person to act on it before a business process changes. An agent may be able to create records, send messages, call software tools, modify data, or trigger downstream steps directly.

For agents, authority becomes a central control concept.

Auditors should understand:

• which systems the agent can access;
• which identity or service account it uses;
• what data it can read;
• what records it can create, change, or delete;
• which tools or APIs it can call;
• which actions require human approval;
• what transaction, value, or volume limits exist;
• how the system determines that a task is complete;
• what happens when a tool fails or returns unexpected results;
• whether actions can be reversed;
• how exceptions are escalated;
• whether material actions are logged in a way that can be reconstructed.

This leads to a second useful rule:

The more an AI system can act, the less sufficient it is to test only the quality of its answers.

Agent assurance must also examine the execution path.

The audit surface expands in four directions

Moving from prompting to RAG to agents changes more than the user experience. It expands the audit surface in at least four directions.

1. Context

With prompting, much of the context is supplied directly by the user.

With RAG, the system selects context from repositories.

With agents, context may also be accumulated dynamically from prior steps, tool results, memory, or system state.

The auditor therefore moves from testing input discipline toward testing information selection and state management.

2. Permissions

A prompt-based assistant may only need access to information intentionally supplied by the user.

A RAG solution requires controlled access to repositories.

An agent may require credentials and permissions across multiple applications.

Identity and access management therefore becomes increasingly important as operational reach expands.

3. Consequence

An unsupported prompt response may remain a draft until a human acts on it.

A RAG answer may carry greater perceived authority because it appears to be grounded in internal documentation.

An agent can potentially convert an incorrect inference into an action.

The right control question is therefore not only, "Can the model be wrong?" but also "What can happen next when it is wrong?"

4. Evidence

Prompting can often be reconstructed from the input, output, and reviewer action.

RAG additionally requires evidence of what was retrieved, from where, and under which version or access conditions.

Agents require evidence of decisions, tool calls, approvals, actions, exceptions, and outcomes.

Observability becomes more important as system autonomy and connectivity increase.

A practical comparison for auditors

What the major frameworks add

The frameworks do not need to be turned into separate checklists. They help answer different parts of the control problem.

NIST: manage the risk across the AI lifecycle

The NIST AI Risk Management Framework and its Generative AI Profile provide a risk-management structure for identifying, assessing, measuring, and managing AI risks across design, development, deployment, and use.

For this comparison, the practical implication is that the risk profile should reflect the actual system architecture. A prompt-only assistant, a RAG implementation, and an agent connected to business applications should not be assessed as though they create the same exposure.

ISO/IEC 42001: govern the surrounding management system

ISO/IEC 42001 is useful because it looks beyond output accuracy. It includes requirements for roles and responsibilities, AI risk assessment and treatment, impact assessment, documented information, operational planning and control, management of changes, monitoring and measurement, and internal audit.

This matters as architecture becomes more connected.

A prompt-based use case may rely heavily on acceptable-use controls, competence, data handling, review, and verification. A RAG solution adds repository governance, retrieval configuration, access, and source maintenance. An agent extends operational control into tools, permissions, approvals, execution, monitoring, and exception handling.

COSO: treat GenAI as an internal-control problem

COSO's 2026 guidance on effective internal control over generative AI reinforces a practical point: rapid GenAI adoption changes workflows and introduces risks that need to be incorporated into the organization's control environment rather than treated as a standalone technology issue.

That perspective is especially relevant for RAG and agents, because many material risks arise in the surrounding process: access, configuration, changes, approvals, monitoring, and the consequences of automated action.

The IIA: audit the organization's AI and govern Internal Audit's own use

The IIA's Artificial Intelligence Auditing Framework provides principles-based guidance for assessing AI-related governance, risk, and controls. Its 2026 prompting guidance also makes the mirror-image responsibility clear: Internal Audit should govern its own use of AI, verify significant outputs, and preserve professional judgment and auditability.

These expectations become more important, not less, when Internal Audit moves from prompting into RAG or agentic workflows.

A RAG-based methodology assistant can systematically spread an outdated interpretation if source governance is weak. An agent connected to an audit-management platform can create a larger operational risk if its permissions, approval rules, or logs are poorly designed.

Internal Audit should therefore ask itself the same question it asks management:

What additional authority did this architecture give the AI, and what additional control did we introduce because of it?

Common mistakes

One mistake is treating every GenAI solution as simply "a chatbot." That hides material differences in source access, permissions, traceability, and operational impact.

Another is assuming that RAG solves hallucination. Retrieval can improve grounding, but retrieval itself can fail, and the model can still misinterpret or overstate the retrieved material.

A third is treating human review as a complete control for agents. Review can be valuable, but it does not compensate for excessive permissions, missing logs, weak approval gates, or actions that occur before a reviewer sees them.

A fourth is confusing technical sophistication with maturity. A more complex architecture is not automatically a better-controlled or higher-value architecture.

Finally, organizations sometimes test the model but ignore the system around it. In enterprise implementations, major control weaknesses may sit in repositories, access rights, APIs, workflow logic, approval mechanisms, monitoring, or change management rather than in the model itself.

Start with the decision, not the architecture

Before choosing prompting, RAG, or an agent, ask:

1. What outcome are we trying to achieve?
2. What information does the AI actually need?
3. Does that information need to be retrieved automatically?
4. Does the AI need authority to act, or is a recommendation sufficient?
5. What happens if the system is wrong?
6. What evidence would demonstrate that the controls operate effectively?

That sequence often leads to a simpler architecture than starting with the technology.

And that is usually a strength, not a limitation.

Prompting, RAG, and agents are not merely three ways to use the same model. They create different control environments.

Prompting changes the instructions and context used for generation. RAG adds a retrieval layer. Agents add operational reach through tools and actions.

For auditors, the practical progression is therefore not "basic to advanced." It is:

generate → retrieve → act

And with each change, the assurance question expands from "Is the output reliable?" toward the broader question:

"Can we trust the information, permissions, decisions, and actions surrounding this AI system?"

Sources

• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
• NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)
• ISO, ISO/IEC 42001:2023 — Artificial intelligence management system
• COSO, Achieving Effective Internal Control Over Generative AI
• The IIA, Artificial Intelligence Auditing Framework
• The IIA, AI Framework Support Tools
• The Institute of Internal Auditors, AI Prompting for Internal Auditors (2026).
• The Institute of Internal Auditors Netherlands and ISACA Netherlands Chapter, Internal Audit in the Age of Generative AI: A Strategic Approach to Implementing AI in Internal Audit (2026).