Auditing the AI Lifecycle: From Idea to Retirement
An AI system can pass every pre-release test and still become a materially different risk six months later. The user population may change. Production data may drift. A new legal requirement may apply. A vendor may update a foundation model. Employees may begin using the tool for decisions that were never part of the approved purpose.
That is why auditing AI cannot be reduced to reviewing a model at one point in time. The assurance question is whether the organization remains in control of the decisions that allow an AI system to be conceived, built, released, changed, kept in production, and ultimately retired.
The practical thesis is simple: lifecycle assurance should follow the chain of decisions and evidence between stages rather than audit each stage as an isolated activity. ISO/IEC 42001 provides management-system requirements and reference controls. The NIST AI Risk Management Framework provides a continuous risk-management lens built around governance, context, measurement, and response. Together, they make the lifecycle auditable without turning it into a rigid checklist.
The biggest lifecycle risks often sit between stages
Organizations naturally focus on development and validation, yet many consequential failures occur at handoffs in accountability: an idea advances without a clear owner, a test exception fails to block release, a supplier changes a component without reassessment, or a system stays in use because nobody established retirement criteria.
ISO/IEC 42001:2023 requires organizations to manage AI risks, assess AI-system impacts where applicable, and plan changes to the AI management system. Annex A includes reference controls covering requirements, design and development, verification and validation, deployment, operation and monitoring, technical documentation, event logging, data, and third-party relationships. One distinction matters for auditors: Annex A is not a universal checklist that every system must implement identically. The standard's risk-treatment process is used to determine necessary controls and to justify inclusion or exclusion in the statement of applicability.
NIST approaches the problem differently. Its Govern, Map, Measure, and Manage functions are not sequential lifecycle phases. They are iterative risk-management functions intended to operate throughout the AI lifecycle. For auditors, that creates four persistent questions at every stage: Who is accountable? What context and risk are being accepted? What evidence shows performance and impact? What action follows from the results?
A strong lifecycle control does more than prove that the system works. It shows why the organization decided the system was allowed to move forward.
Seven assurance gates from idea to retirement
There is no single lifecycle sequence that fits every organization. A seven-gate model, however, gives auditors a practical way to organize assurance around decisions rather than technology labels.
| Stage | Audit question | Expected evidence | Useful procedures |
|---|---|---|---|
| Approval | Should AI be used for this purpose, and at what level of risk? | Business case, accountable owner, intended purpose, risk classification, initial impact assessment, legal requirements, go/no-go decision | Trace approval to defined criteria; test segregation of responsibilities; challenge critical assumptions and whether non-AI alternatives were considered |
| Design | Do the requirements translate objectives and risks into testable criteria? | Functional and control requirements, intended users, use limitations, data needs, human oversight, acceptance criteria | Map requirements to risks and impacts; review foreseeable misuse; confirm metrics and thresholds can actually be tested |
| Development | Was the solution built with adequate traceability and control? | Versions, data provenance and quality, design decisions, experiment records, access controls, tests, exceptions | Sample versions and changes; trace data to sources; evaluate exceptions; involve specialists when the audit objective requires deep technical validation |
| Deployment | Does the evidence support releasing this system into the real operating environment? | Verification and validation results, release criteria, approvals, production configuration, user guidance, monitoring and rollback plans | Reperform selected release criteria; reconcile approved and deployed versions; confirm controls and monitoring are active before release |
| Monitoring | Is the system still operating within acceptable limits? | Performance and risk metrics, alerts, incidents, complaints, human overrides, drift indicators, support records, periodic reviews | Analyze trends; inspect exceptions and incidents; test alerting; assess whether changes in context, data, or use trigger reassessment |
| Change | Does the modification reopen the right risks and approvals? | Change request, materiality classification, impact analysis, revalidation, approval, updated documentation and inventory | Sample model, data, prompt, vendor, or purpose changes; test materiality criteria; verify material changes route back to earlier lifecycle gates |
| Retirement | Can the system be decommissioned without creating new risk? | Retirement decision, dependency map, migration plan, record retention, data treatment, access revocation, inventory update | Test upstream and downstream dependencies; review legal and forensic retention; verify interfaces and credentials are disabled; confirm transition to replacement processes |
The model establishes auditable decision gates, not a mechanical checklist. Assurance depth should increase with the system's impact, complexity, and exposure.
The strongest evidence is traceability across decisions
An auditor should be able to take an approved statement from the beginning of the lifecycle and follow it into live operation. If the approved use case says a human will always review an AI recommendation before action is taken, that expectation should appear in the requirements, interface design, testing, operating procedures, and ultimately in records showing whether human oversight actually occurred.
The evidence chain can be expressed as:
purpose → requirements → risks and impacts → controls → testing → approval → monitoring → change → retirement.
A broken link is an audit signal. A production metric that was never tied to a risk; a release approval with no acceptance threshold; a supplier update with no revalidation evidence; or a retired application with active credentials are governance failures even when the model's technical scores look strong.
Third-party AI makes this chain even more important. ISO/IEC 42001 includes reference controls for allocating lifecycle responsibilities among the organization, suppliers, customers, and other parties. Internal Audit does not need unrestricted access to a vendor's model to provide useful assurance, but it does need to establish what responsibility the organization retains, what evidence it receives from the supplier, and what happens when the service changes.
ISO/IEC 42001 and NIST play different roles
The two frameworks are complementary, but they should not be treated as interchangeable.
| Framework | What it contributes | How auditors can use it |
|---|---|---|
| ISO/IEC 42001:2023 | Requirements for an AI management system plus reference controls covering risk, impact, resources, lifecycle, data, use, and third parties | Use applicable requirements, internal policies, and controls selected in the statement of applicability as evaluation criteria; distinguish requirements from implementation guidance |
| NIST AI RMF 1.0 | A voluntary, outcome-oriented structure for managing AI risk and trustworthiness through Govern, Map, Measure, and Manage | Use it to structure risk questions, governance expectations, and procedures; do not treat its functions as lifecycle phases or the Playbook as a mandatory checklist |
The distinction is particularly useful at the end of the lifecycle. Govern 1.7 of the NIST AI RMF explicitly addresses safe decommissioning and phasing out of AI systems. ISO/IEC 42001's reference controls focus heavily on requirements and development through operation and monitoring, while decommissioning also appears in its resource and risk considerations. For an audit program, the implication is straightforward: retirement needs criteria, accountability, and evidence, not merely a technical deletion step.
Four common failures in lifecycle audits. Four patterns weaken lifecycle assurance: auditing only the launch; using the same depth for every system; treating material change as routine maintenance; and ignoring retirement. A risk-based approach should scale testing and monitoring, force relevant changes back through earlier gates, and establish decommissioning criteria for dependencies, retention, access, and transition.
An audit program that tests decisions, not document presence
A useful engagement objective could be: determine whether AI systems are governed throughout their lifecycle through stage criteria, accountable ownership, evidence, monitoring, and change and retirement controls proportionate to risk.
From that objective, Internal Audit can select a sample of AI systems and:
- reconstruct the original approval and compare the approved purpose with current use;
- trace identified risks and impacts into requirements, controls, and test criteria;
- reconcile the approved version with the version actually deployed;
- assess whether operating metrics have thresholds, owners, and defined responses;
- sample recent model, data, prompt, supplier, or purpose changes and verify whether they triggered revalidation, renewed approval, or inventory updates;
- review a retired or soon-to-be-retired system for dependencies, evidence retention, access removal, and transition controls.
The final question is not whether the organization has a document for every stage. It is whether it can demonstrate, through coherent evidence, why the system was allowed to proceed, why it remains acceptable in production, when it must be reassessed, and how the decision to stop it is controlled.
That is the value of lifecycle auditing: turning AI governance into a series of verifiable decisions from the first business idea to the final records retained after decommissioning.