EN
Español English

Auditing Model Risk Without Being a Modeler

A model can fail because its technical design is weak. It can also fail because no one owns the decision it supports, an independent validation never happened, operating limits were not documented, or performance deteriorated without triggering action. For internal audit, that distinction matters: a large part of model risk can be audited without rebuilding the model.

Two reactions are common and both are unhelpful. One is to avoid AI-related model risk because it feels too technical. The other is to behave as if a generalist audit team can substitute for a model validator. A better approach is to audit the system of accountability around the model: who owns it, how it is approved, how validation is governed, what limitations are accepted, how performance is monitored, and what happens when the model changes or fails.

The Global Internal Audit Standards support that boundary. Standard 3.1 requires internal auditors to possess or obtain the competencies needed for their responsibilities. Standard 13.5 requires the engagement team to consider whether human, financial, and technological resources are appropriate and sufficient for the engagement objectives. The implication is not that every auditor must become a data scientist. It is that scope and resourcing must match the assurance question.

Do not confuse auditing validation with performing validation

Technical model validation can involve statistical testing, sensitivity analysis, challenge of assumptions, benchmarking, stability analysis, fairness testing, and out-of-sample performance review. Those activities may require specialist expertise. Internal audit can still assess whether the validation process is credible without reperforming all of them.

A generalist can ask whether validation occurs before production use, whether the validator is sufficiently independent, whether acceptance criteria are defined in advance, whether limitations are documented, whether unresolved findings are visible to the approving authority, and whether material changes trigger revalidation. Those are governance, control, and evidence questions.

The IIA's Auditing Model Risk Management, 2nd Edition is recommended Global Guidance, not a mandatory requirement equivalent to the Standards. It helps internal auditors structure assurance over model risk management, including governance, compliance, and reliability. The IIA Artificial Intelligence Auditing Framework provides a broader bridge between AI strategy, governance, management, risk, controls, and internal audit.

Four questions a generalist auditor can answer

Area What a generalist can assess Useful evidence When specialist support is warranted
Governance and ownership Whether the model is inventoried, risk-classified, assigned to accountable owners, and subject to defined approval authority Model inventory, RACI, policy, committee minutes, classification criteria When complex architecture or third-party dependencies obscure which component creates the risk or who controls it
Validation and approval Whether validation occurs before use, is sufficiently independent, uses defined criteria, and informs the go-live decision Validation report, criteria, findings, exceptions, approval record When the audit objective requires statistical reperformance, challenge of mathematical assumptions, or specialist methodology review
Monitoring and change Whether metrics, thresholds, drift alerts, revalidation triggers, and withdrawal rules exist and operate Dashboards, logs, alerts, tickets, version history, revalidation evidence When the team must determine whether a metric or threshold is technically/statistically appropriate
Use and oversight Whether users understand limitations, human review is meaningful where required, and third-party dependencies are controlled Operating procedures, user guidance, review evidence, contracts, SLAs, incident records When advanced fairness, explainability, robustness, or adversarial-security testing is needed

This is the practical boundary: internal audit can determine whether the management system works without claiming to have mathematically proven that the model is optimal.

Audit the validation process, then challenge the decision it supports

The NIST AI Risk Management Framework 1.0 is a voluntary framework organized around Govern, Map, Measure, and Manage. Its Measure function is especially useful for assurance because it emphasizes testing before deployment and during operation, context-appropriate performance criteria, documentation of limitations, and involvement of assessors who are sufficiently independent from front-line development.

Consider a fraud model with a high false-positive rate. Internal audit does not necessarily need to rebuild the classifier. It can ask who defined the acceptable false-positive level, what business impact was assessed, what evidence supported approval, whether alternative thresholds were considered, and what breach level triggers review. Those questions can expose a serious model-risk weakness even when no auditor writes a line of model code.

As of September 2026, NIST AI RMF 1.0 remains the current public version while NIST works on a revision. Audit programs should therefore record the version and date of the criteria they apply rather than referring generically to “NIST requirements.”

Monitoring is where model risk becomes operational

A model that passed validation can become unreliable later. Inputs change. Customer behavior shifts. Business processes are redesigned. New regulations alter acceptable outcomes. Upstream data pipelines change. Even a model that does not continuously learn can drift away from the conditions under which it was approved.

ISO/IEC 42001:2023 establishes requirements for an AI management system and embeds monitoring, measurement, performance evaluation, controlled change, and improvement in the management cycle. Its implementation guidance also recognizes that performance may change because of data or concept drift and that monitoring can identify the need for retraining or other corrective action.

For a generalist auditor, the useful test is not “does a dashboard exist?” It is whether management has defined what is monitored, by whom, at what frequency, against what thresholds, and with what required response. An effective design should connect deteriorating performance to action: investigation, restriction, revalidation, adjustment, or withdrawal from production.

The IIA's AI risk and control support material uses the same practical logic: monitor performance, stability, fairness indicators and data quality; define thresholds; link breaches to actions; retain evidence of revalidation and remediation. That is enough to create substantive audit procedures without turning the engagement into a data-science project.

Ownership matters more than algorithmic sophistication

“Who owns the model?” often produces an incomplete answer. A data-science team may build it, a vendor may host it, a business unit may act on its output, and risk or compliance may define oversight controls. None of those arrangements removes management accountability for the business decision the model supports.

The IIA Three Lines Model helps preserve that distinction: management owns and manages risk; second-line roles provide specialist support, monitoring, and challenge; internal audit provides independent assurance and advice without assuming management responsibility. COSO ERM can add the enterprise-risk lens by connecting model risk to risk appetite, strategy, and performance, but it is not a technical model-validation standard.

A useful audit trail should therefore be traceable end to end: intended purpose → accountable owner → risk classification → approval criteria → validation → usage limitations → production monitoring → incidents and changes → decision to continue, adjust, restrict, or retire.

Knowing when to bring in a specialist is part of audit competence

A generalist auditor can go far by understanding the model's purpose, business consequences, governance, and expected evidence. The boundary appears when the audit conclusion depends on a technical assertion the team cannot support competently.

Specialist support may be necessary to statistically revalidate a model, review model code or architecture, design advanced fairness testing, assess adversarial robustness, or determine whether a complex metric is suitable for a specific population. Internal audit still owns the assurance process: framing the question, evaluating the specialist's work, integrating the evidence, applying professional skepticism, and communicating the conclusion.

That is the core message: you do not need to be the modeler to audit model risk well. You need to know which questions a generalist can answer, what evidence should exist, and where specialist evidence becomes necessary.

Audit competence is not demonstrated by rebuilding the model. It is demonstrated by knowing what must be governed, evidenced, challenged, and escalated.

Sources