How to Test AI Inventory Completeness

Management says the organization has 37 AI systems in its register. The auditor asks a simple question: how do we know there is not a thirty-eighth?

If the answer is “every business unit is required to register its AI,” the list may show what people declared. It does not yet show that the population is complete. An unregistered use can sit inside an employee-paid subscription, a newly enabled feature in an existing SaaS platform, a cloud service consumed by a technical team, or a public generative AI tool used without a formal purchase.

The central audit idea is straightforward: you do not test AI inventory completeness by inspecting the inventory. You test it by trying to find AI outside it.

Define what “complete” means before testing it

Completeness needs an auditable scope. An AI inventory does not necessarily need every algorithm that exists inside the enterprise if governance uses a narrower, documented definition or risk threshold. It does need to consistently capture the systems and uses that the organization says are subject to governance, risk assessment, approval, or monitoring.

Before extracting data, clarify three points:

1. What qualifies as AI? Does the definition cover internally developed models, AI embedded in SaaS, generative assistants, model APIs, public tools, and agents?
2. What is the unit of inventory? Is the organization registering a product, model, use case, or a combination?
3. What may legitimately be excluded? If low-risk embedded features are outside scope, the exclusion logic should be documented and applied consistently.

ISO/IEC 42001:2023 provides a management-system framework for governing the development and use of AI. Its reference controls include identifying and documenting relevant AI resources across lifecycle activities. The standard does not tell internal auditors to inspect corporate-card transactions or SaaS catalogs. Those are audit procedures designed to test whether the organization’s identification mechanism works in practice.

The NIST AI RMF Core is explicit that mechanisms should exist to inventory AI systems. Its Playbook describes the inventory as a way to obtain a holistic view of organizational AI assets. For internal audit, that makes the key assurance question less about the appearance of the register and more about the evidence supporting its coverage.

Build an independent discovery population

A common testing error is to start with the official inventory and select a sample. That can test the accuracy of sampled records. It cannot, by itself, test the completeness of the population.

Completeness testing needs sources that originate outside the process maintaining the register. Different sources expose different blind spots.

The procedure should not become a keyword search for “AI.” Product names change, and AI is increasingly delivered as a feature inside software that was originally purchased for another purpose. Effective discovery combines vendor patterns, service descriptions, spend categories, cloud services, technical records, and business context.

Reconcile sources as you would search for missing transactions

A practical completeness test can be run as a population reconciliation.

1. Freeze the inventory at a point in time. Retain the version being audited, including identifiers, owners, vendors, use cases, and lifecycle status.

2. Extract candidates from independent sources. Treat them as leads, not findings. A merchant name or product description is not enough to conclude that an unregistered AI use exists.

3. Normalize names and relationships. The same product may appear under the vendor’s legal entity in procurement, a different domain in SSO, and an abbreviated descriptor on a card transaction. Normalization prevents one tool from becoming three exceptions.

4. Match the candidates to the official register. Classify each as a confirmed match, potential unregistered item, out-of-scope item, or false positive.

5. Investigate unmatched candidates. Confirm with the business owner what the tool does, who uses it, what data it processes, whether it includes AI, and whether it meets the organization’s registration criteria.

6. Test some matched records as well. A register may be reasonably complete but inaccurate. Sample reconciled items to confirm ownership, purpose, vendor, lifecycle stage, and risk classification.

This approach also fits the Global Internal Audit Standards. Standard 13.6 requires the engagement work program to identify the tasks, methodologies, analytical procedures, and tools needed to achieve engagement objectives. Standard 14.1 requires information used for analysis and evaluation to be relevant, reliable, and sufficient. Independent-source reconciliation strengthens reliability and reduces the risk of using the population under audit as the sole evidence that the population is complete.

Make surveys behavior-based, not AI-based

A business survey can uncover tools that never appear in procurement or system inventories, but asking “Do you use artificial intelligence?” produces inconsistent evidence. Some employees may not know an embedded feature uses AI. Others may label ordinary automation as AI.

Better questions describe activities:

• Which tools does your team use to generate, summarize, translate, or review content?
• Which systems classify, score, recommend, predict, or prioritize people, transactions, or cases?
• Which AI-enabled features have been activated inside applications your team already used?
• Which public tools or individually purchased accounts are used for work activities?
• Which pilots are operating even if they have not entered formal production?

Then reconcile the responses with SaaS, expense, procurement, and technical evidence. An uncorroborated survey is another self-reported list, not proof of completeness.

Measure whether the discovery process is improving

Internal audit should not become the permanent owner of the AI inventory. The objective is to assess whether governance has repeatable mechanisms to detect new uses and update the register.

Three measures can help make the conclusion more evidence-based:

• Unregistered discovery rate: the proportion of validated AI uses found outside the inventory during testing.
• Exception source concentration: which discovery channels, business units, or spend categories generate the most unregistered uses.
• Registration lead time: how long it takes from discovery to registration, assessment, or documented closure.

There is no universal percentage that proves completeness. The conclusion depends on risk, scope, source coverage, and follow-up quality. But repeated discoveries from the same channels are a strong signal: if expense data or SaaS records continually surface valid AI uses that the governance process missed, the identification control is not operating consistently.

A complete inventory is a process, not a snapshot

Even a perfectly reconciled register can become stale quickly. Vendors add AI functionality to existing products, employees try free tools, new cloud services are consumed, and pilots move into operational use.

The stronger audit question is therefore not only “Is the inventory complete?” It is: what mechanisms make it likely that a new AI use will be detected, assessed, and registered before it becomes an unmanaged exposure?

That distinction matters. A polished register can still depend on voluntary declarations. A stronger control environment uses multiple detection points, assigns accountability, investigates exceptions, and learns from every legitimate use discovered outside the register.

Completeness is not asserted. It is earned through independent discovery and reconciliation.

Sources

• ISO, ISO/IEC 42001:2023 — Information technology — Artificial intelligence — Management system
• The Institute of Internal Auditors, Complete Global Internal Audit Standards
• NIST, AI Risk Management Framework — Core
• NIST, AI RMF Playbook — Govern