Who Is Accountable When AI Makes Decisions?
An AI system recommends rejecting a supplier. The employee responsible for the process accepts the recommendation. Weeks later, the organization discovers that it relied on incomplete information. Technology points to the vendor that supplied the model. The vendor says the client configured its use. The business notes that a person approved the result. The employee says they followed the system.
That is not only a model failure. It is an accountability failure: no one can clearly explain who had authority to decide, who was expected to challenge the output, and who had to answer for the consequences.
Good AI governance starts there. A system can generate a prediction, classification, or recommendation and may automate part of a decision. It does not replace the people and organizational roles that authorize, use, oversee, change, and monitor that system. Legal liability depends on jurisdiction and facts; organizational accountability should be designed before an incident forces the question.
Automation can move a decision. It cannot move the obligation to answer for it.
AI is not a seat on the organization chart
Saying “the AI made the decision” may describe what happened technically, but it is a weak governance explanation. Auditors should separate three layers: the system output — a score, recommendation, or classification; the business decision — approve, reject, block, or escalate; and accountability for that decision, including authority to accept it, change it, stop it, and explain its effects.
The IIA's refreshed Three Lines Model helps keep those layers clear. Management retains responsibility for managing risk and operating processes and controls. Second-line roles provide specialist support, monitoring, and challenge. Internal Audit provides independent assurance and advice without taking management responsibility or decision-making authority.
That distinction matters. Assessing AI governance is not the same as owning it. Internal Audit can challenge responsibilities and controls, but should be cautious about approving models, accepting residual risk, or deciding whether an AI system may operate when it may later need to provide assurance over those same decisions.
Govern the decision before the model
A responsibility matrix can help, but the first question should not be “Who owns the model?” A better question is: What business decision can this system influence, and who has authority over that decision?
| Role | Accountability question | Useful evidence |
|---|---|---|
| Board and senior management | Who sets boundaries, risk appetite, and escalation expectations? | Policy, mandates, minutes, risk acceptance criteria |
| Process or system owner | Who answers for intended use and performance? | AI inventory, use-case approval, controls, monitoring |
| Human overseer or decision-maker | Who can challenge, override, stop, or escalate? | Instructions, training, review and exception records |
| Provider and third-party owner | Which duties sit with each party? | Contracts, SLAs, documentation, change and incident notices |
One person may legitimately hold several roles. The problem arises when a critical activity has no owner, several parties are “responsible” but no one has final authority, or the named owner lacks the information and power to act.
A RACI-style matrix can be useful, although it is not a universal requirement. What matters is whether the organization can show who approved the use, who accepted the risk, and who can order remediation or suspension.
ISO/IEC 42001 makes role allocation auditable
ISO/IEC 42001:2023 treats accountability as a management-system issue. Clause 5.3 requires top management to ensure that responsibilities and authorities for relevant roles are assigned and communicated. It also requires responsibility and authority to be assigned for ensuring that the AI management system conforms to the standard and for reporting its performance to top management.
Annex A reinforces that logic through reference control A.3.2 on AI roles and responsibilities. It is not an automatic checklist in which every listed control is implemented identically by every organization. ISO/IEC 42001 uses a risk-based approach, and organizations determine needed controls through risk treatment and their statement of applicability.
The implementation guidance in Annex B points to areas where responsibilities may need to be explicit, including risk management, impact assessment, security, privacy, development, performance, data quality, human oversight, and supplier relationships. For an auditor, “the AI team is responsible” is rarely enough to show those duties are actually covered.
The standard also requires designated management approval for the AI risk treatment plan and acceptance of residual AI risks. Mature accountability therefore does more than assign technical tasks; it establishes decision rights over risk.
The EU AI Act gives human oversight operational meaning
The EU AI Act allocates duties across defined roles, including providers and deployers. For certain high-risk AI systems, Article 14 makes human oversight more substantive than placing a person at the end of a workflow.
Oversight measures are intended to enable assigned people to understand relevant capabilities and limitations, monitor anomalies, remain alert to automation bias, interpret outputs, decide not to use or to override or reverse an output, and intervene or stop the system where appropriate. Article 26 requires deployers of high-risk systems to assign oversight to people with the necessary competence, training, authority, and support.
The audit implication is simple: a person is not an effective control merely because the workflow contains a human approval step. If the reviewer lacks time, information, competence, or authority to challenge the system, oversight may be ceremonial.
Article 25 also addresses responsibilities along the AI value chain and describes circumstances in which a distributor, importer, deployer, or other third party can be treated as the provider of a high-risk AI system. It also provides for written agreements in specified circumstances to support the information and assistance needed between providers and third parties.
Timing matters. As of September 2026, the AI Act is generally applicable with several exceptions, while high-risk-system rules have extended transition periods. The European Commission's current timeline states that Annex III high-risk use cases will be subject to those rules from 2 December 2027, while high-risk systems embedded in Annex I regulated products will follow from 2 August 2028. Auditors can use the role-allocation logic as a governance benchmark now, but should not report premature legal noncompliance before the relevant rules apply to the system being audited.
A vendor relationship splits duties; it does not erase yours
An accountability gap often appears when an organization treats purchased AI as something the vendor “owns.” The supplier may control the foundation model, infrastructure, or certain updates. The customer still decides how the system is used: its purpose, input data, authorized users, process integration, thresholds, exceptions, and actions taken from outputs.
ISO/IEC 42001 addresses third-party relationships in its control guidance. Organizations should understand and document the parties involved across the AI system life cycle and determine their responsibilities, while considering supplier risk, needed documentation, ongoing monitoring, and corrective action when supplied components do not perform as expected.
For Internal Audit, checking that a contract exists is not enough. Depending on risk, the arrangement should make it possible to answer: Who notifies the customer of model changes? Who investigates an incident? What evidence can the customer obtain? How are known limitations documented? What happens after a critical vendor change? How can the service be suspended or exited?
The principle is straightforward: outsourcing technology is not the same as outsourcing accountability for how the organization uses it.
What Internal Audit should test
The strongest accountability test is to trace real decisions end to end and determine whether the authority described on paper also existed in practice. A fundamental work program can include five tests:
- Reconstruct one material decision. Identify the AI output, who or what converted it into action, the evidence considered, and the final outcome.
- Trace decision rights. Verify who could authorize the use case, accept residual risk, approve changes, and stop the system.
- Test human oversight. Sample reviews, overrides, and escalations to confirm that people had sufficient information, competence, and real authority.
- Follow a change. Select a model, configuration, or vendor update and determine whether it triggered reassessment, testing, and reauthorization where required.
- Compare contract with operation. Verify that obligations for documentation, changes, incidents, support, and evidence actually operate.
The red flags are familiar: everyone is “responsible” but no one has final authority; the business assumes Technology owns the risk; the human reviewer cannot override the output; a vendor change does not trigger reassessment; or Internal Audit participates in approvals it will later be asked to audit.
The final question is not “Who built the algorithm?” It is: Who can explain what decision the AI is allowed to influence, who approved that use, who can challenge or stop it, who monitors its effects, who accepts residual risk, and how those responsibilities change when a third party enters the chain?
If the organization cannot answer with evidence, it does not yet have an accountability chain. It has a chain of assumptions.