Data Leakage Through Generative AI Tools

In 2023, Samsung restricted employee use of public generative AI tools after staff uploaded sensitive information, including source code, to ChatGPT. The incident did not require a sophisticated cyberattack. An employee moved company information into an external service because the tool was useful for completing a task.

That pattern remains one of the simplest ways for organizations to lose control of information. The risk does not begin when a model produces a wrong answer. It begins when someone decides what to paste, what to upload, which application to use, and whether that content is allowed to leave the organization’s controlled environment.

The practical lesson is that a policy saying “do not share confidential information with AI” is not enough. Effective prevention requires a control chain connecting data classification, acceptable-use rules, approved platforms, technical safeguards, user awareness, and monitoring.

Data can leak before the model produces an answer

“AI data leakage” is often understood as a model revealing information on its own. That is a real risk, but it is only one pathway.

Organizations should think about at least four routes:

Exposure route	Example	Control question
User input	An employee pastes code, customer data, or a contract into an unapproved tool.	Is this information permitted to leave the organization?
Provider processing	Content is processed, retained, or replicated outside the organization’s own control environment.	Have retention, access, and data-use terms been assessed?
System output	A model reproduces sensitive, personal, or restricted-source information.	Are outputs monitored or filtered where the risk warrants it?
Integrations and agents	An AI application uses connectors to access email, repositories, files, or enterprise systems.	Are permissions constrained to the minimum necessary?

The NIST Generative AI Profile treats data privacy risk broadly, including leakage, unauthorized use, disclosure, and de-anonymization of personally identifiable or sensitive data. Its suggested actions include maintaining approved GAI technology and provider lists, updating acceptable-use policies, and monitoring for sensitive-data exposure.

Samsung illustrates the first route: the user created the exposure by entering the data. The March 2023 ChatGPT incident illustrates a different layer of risk. OpenAI reported that a software bug allowed some users to see other users’ chat-history titles and may have exposed limited payment information for a small subset of active subscribers. The lesson is not that one platform is inherently unsafe. It is that third-party risk remains even when employees follow internal rules.

A policy without classification and approved platforms is difficult to enforce

Before opening an AI tool, an employee should be able to answer two questions: “What kind of data is this?” and “Where is it allowed to be processed?”

A simple classification scheme can be translated into practical rules. Public information may be permitted in authorized tools with limited additional controls. Internal information may require an approved enterprise environment. Confidential or restricted information may require prohibition, anonymization, explicit authorization, or a specific architecture that keeps the information within agreed controls.

The important point is to connect labels to decisions. For each information class, organizations should define at least:

which tools or environments are approved;
whether prompts, file uploads, code, images, or recordings are permitted;
whether personal data, trade secrets, or customer information require additional conditions;
which internal connectors and knowledge sources the application may access;
how long information may be retained and who can access it;
what exceptions require approval and how they are recorded.

The IIA’s AI Framework Support Tools, including AI Prompting for Internal Auditors, apply the same logic to the internal audit function itself: AI use should align with organizational governance, approved tools, data-handling requirements, and internal control expectations. This creates an important credibility test. Internal Audit cannot expect the business to govern GenAI while placing audit information into tools that fall outside those same guardrails.

Different frameworks point to the same control architecture

The relevant frameworks do not have the same authority or purpose, but they converge on a similar operating model.

The Global Internal Audit Standards are mandatory professional requirements under the IPPF. Standard 5.2, Protection of Information, requires internal auditors to respect confidentiality, privacy, and ownership of information, comply with applicable requirements, and manage the risk of inadvertent exposure or disclosure. For internal auditors, protecting information when using GenAI is therefore not merely a technology recommendation; it is directly connected to professional conformance.

The NIST AI RMF: Generative AI Profile is voluntary. It provides highly actionable risk-management practices: inventory third parties with access to organizational content, maintain approved-technology lists, incorporate privacy and security into vendor due diligence, update acceptable-use policies, and establish monitoring and incident-response processes for third-party GAI services.

ISO/IEC 42001:2023 is a certifiable management system standard. It does not prescribe a specific technology. Instead, it provides requirements for policy, roles, risk assessment and treatment, operational controls, monitoring, internal audit, and continual improvement. Its AI data and resource controls help connect tool use to provenance, data categories, intended use, retention, privacy, and security.

COSO’s 2026 publication Achieving Effective Internal Control Over Generative AI applies the five components of the Internal Control—Integrated Framework to GenAI. It emphasizes clear boundaries, GenAI acceptable-use policies that identify prohibited data, accountability for misuse, and monitoring that keeps pace with rapidly changing AI configurations and use cases.

The point is not to adopt every framework as a checklist. A more useful question is: Does the organization have a control design that connects policy, technology, and actual user behavior?

What Internal Audit should test

An audit of GenAI data leakage should go beyond confirming that a policy exists. It should determine whether the rules can be enforced and whether deviations can be prevented or detected.

Audit objective	Useful evidence	Possible test	Red flag
Identify actual GenAI use	AI inventory, SSO records, proxy logs, procurement, expenses, browser extensions	Reconcile the inventory to independent sources	Tools appear in logs but not in the inventory
Validate data-handling rules	AI policy, classification standard, permitted-use matrix	Trace sample data types to the applicable rule	“Do not share sensitive data” without an operational definition
Confirm approved platforms	Approved-provider list, risk assessments, contracts, configurations	Review a sample of tools from approval through deployment	Widespread use of personal or free accounts
Evaluate preventive controls	DLP, domain restrictions, CASB/SSE, upload controls, access permissions	Perform controlled test cases or inspect technical evidence	The control relies only on user memory
Evaluate monitoring	Usage logs, alerts, exception reports, investigation records	Trace alerts through investigation and closure	No visibility into uploads, connectors, or high-risk usage where monitoring would be proportionate
Review training and behavior	Role-based training, campaigns, attestations, disciplinary cases	Sample users and test understanding	Generic training without examples tied to real data
Assess incident response	Procedures, tickets, root-cause analysis, corrective actions	Trace an incident from detection to remediation	The event is treated only as “user error” and the underlying control gap remains

Monitoring should remain proportionate. Capturing every prompt may be unnecessary, intrusive, or create a new repository of sensitive information. The control objective is not maximum collection. It is enough visibility to detect material misuse while avoiding unnecessary additional exposure.

The goal is not to ban AI but to make data flows visible

A blanket ban on public tools may reduce one exposure route, but it rarely solves the whole problem. Employees still seek productivity, AI features are increasingly embedded in existing applications, and new services emerge faster than traditional policy cycles.

A more durable control model starts from a different premise: the organization should know which data can enter which systems, under what conditions, with what permissions, and with what evidence of monitoring.

Internal Audit should apply that discipline externally and internally. If auditors use GenAI to summarize documents, develop test procedures, analyze information, or draft work products, that use should occur within a defined methodology and through authorized tools. The function’s credibility in auditing AI governance begins with governing its own use of AI.

Generative AI data leakage is not an entirely new risk. It is a new route for familiar risks involving confidentiality, privacy, third parties, access, and human behavior. What changes is the speed with which a user can move information into an external environment and how easily that action can bypass traditional controls. Effective governance therefore does not sit in one prohibition. It sits across the full chain from classifying the data to monitoring its use.