Generative AI tools like ChatGPT have drastically changed the way we work. They’ve transformed how people write code, draft emails, summarize contracts, and get through their workday faster.
This change has been rapid. So rapid, in fact, that many organizations still don't have a clear picture of how much sensitive data their employees are pasting into ChatGPT. Most have no formal policy, let alone any detection in place. This is a huge cybersecurity problem.
We’re not talking about a hypothetical risk here. Generative AI data leaks have already happened to some of the world's largest and most security-conscious companies, including Amazon, Samsung, and many more.
In this article, we’ll examine real-world ChatGPT risk incidents, unpack the technical mechanisms behind AI data leaks, and lay out a practical framework that IT and security leaders can use to keep generative AI use safe.
But first, let’s get specific: what are ChatGPT risks, exactly?
Why ‘ChatGPT risk’ is actually three separate problems
When people talk about ChatGPT risks, they're usually conflating three distinct problems.
Problem 1: Input risk
The first is input risk: an employee pastes sensitive data into a generative AI tool like ChatGPT to help complete a task. This could be a snippet of source code, a client contract, or a customer record. That data technically now lives outside your company's control, on a third party's server, and depending on account settings, it may be used to train future versions of the model.
This is the risk most security teams already have some visibility into, usually because it shows up in the same channels (browser traffic, SaaS logs, endpoint activity) that a CASB or DLP tool is already monitoring for unauthorized apps or activity.
The problem is that most DLP policies were written with email attachments and file uploads in mind, not chat interfaces where employees can paste an entire contract in seconds with no file extension, no upload event, and nothing that looks like a traditional exfiltration pattern to flag.
Problem 2: Output risk
The second is output risk: the possibility that the model itself can be manipulated into regurgitating memorized training data to an entirely different user, including sensitive or confidential information. This is a newer and less understood risk, and it's one most security teams haven't factored into their thinking yet.
Both of these problems fall under the broader umbrella of shadow AI: the AI-specific subset of shadow IT that IT and security leaders need to be actively tracking, not just assuming employees self-regulate.
Input-side controls (DLP, CASB, acceptable-use policy) can meaningfully reduce your exposure to input risk, but they do nothing to control output risk. Any complete framework has to address both, rather than treating one DLP rollout as a finished project.
Problem 3: Model hallucinations
There's also a third, related risk: employees trusting hallucinated or incorrect ChatGPT output in their actual work, the way some lawyers have famously done when preparing legal briefs.
This is a real and significant problem, but the solutions are distinct from input and output risk. Namely, companies need to train their teams to second-guess everything they find via generative AI, and only rely on information they’ve verified independently.
Now, let’s look at some real-world examples of ChatGPT risks in action.
Case study: how employee input becomes exposure (Amazon)
In January 2023, Amazon had to formally warn its staff about their use of ChatGPT after a legal team member flagged a troubling pattern. According to internal Slack messages (later obtained by Business Insider) an Amazon lawyer told employees the company had already spotted ChatGPT-generated text that closely resembled the company's own internal data.
Amazon’s biggest concern here was that these prompts could become training data for a future iteration of the model, and they didn't want ChatGPT's output to eventually include or resemble its own confidential code.
This case is a lesson for every company. The cause of the leak was Amazon engineers using ChatGPT as a coding assistant, pasting in existing internal code with the hope of improving or debugging it. The use of AI, as well-intentioned and productivity-driven as it was in this case, was still dangerous because there were few guardrails around it. Even with additional system protections in place today, this can still be a risk.
Case study: companies restricting ChatGPT (Goldman Sachs, Samsung, Apple)
By mid-2023, a wave of companies had either restricted ChatGPT or banned it outright. These included Goldman Sachs, Apple, Verizon, JPMorgan Chase, Citigroup, Bank of America, and others. Privacy was the primary concern cited across nearly all of these companies for these restrictions, largely tied to the fact that ChatGPT could use conversation data to help train future models.
In April 2023, Samsung engineers accidentally leaked confidential internal source code and recordings of internal meetings. This happened when someone pasted them into ChatGPT to get help with debugging and summarizing.
Once that story broke, Apple quickly moved to restrict both ChatGPT and GitHub Copilot for employees, citing similar fears about data leakage. Goldman Sachs took a similar path, blocking ChatGPT through its standard third-party software controls, while simultaneously building its own internal generative AI tools to give employees a sanctioned way to get the productivity benefits without the exposure.
Case study: how ChatGPT can be manipulated into leaking training data (Google DeepMind, Cornell)
In late 2023, researchers from Google DeepMind, Cornell, and several partner universities found they could get ChatGPT to regurgitate memorized training data simply by prompting ChatGPT to repeat a single word like "poem" forever.
After a few hundred repetitions, the model would break from its expected behavior and start generating fragments of its original training data, including real people's email signatures and contact information. Some trigger words were far more effective than others. For example, the researchers found that prompting the model to repeat "company" caused it to leak training data roughly 164 times more often than a neutral control word.
Using only $200 worth of API queries, the researchers were able to extract more than 10,000 unique, verbatim memorized training examples, and they explicitly noted that a more resourced adversary could likely extract far more. This is known as a divergence attack: a technique that intentionally pushes a model outside its aligned behavior until it starts outputting raw, unfiltered fragments of the material it was trained on, rather than a polished chatbot response.
Technically speaking, any data an employee has ever typed into an open version of ChatGPT could potentially be extracted by someone outside your organization at some point in the future. OpenAI has patched the specific technique researchers disclosed, but the broader category of memorization and extraction risk in large language models hasn't gone away, and it's an active area of adversarial research, not a one-time bug.
With all that in mind, let’s dig into a practical framework for IT and security leaders to encourage the practical use of AI tools like ChatGPT.
Learn more: AI Data Leaks: How They Happen, and How the Right DLP Tool Can Help
A practical 7-step framework for IT and security leaders to manage ChatGPT risks
Most organizations don't need to go so far as to ban ChatGPT just to take control of these risks. What they do need is a working framework their teams can actually follow.
Here's a seven-step checklist to help you get there:
- Classify before you ban: Start with an inventory of what kinds of sensitive data are most likely to end up in employee prompts in your organization: PII, source code, financials, contracts, M&A information. You can't build sensible policy or detection rules until you know what you're protecting.
- Choose your posture deliberately: The companies we mentioned earlier took three broad approaches: a hard ban, offering an approved alternative, or allowing tool usage conditional on data loss prevention (DLP) monitoring. Pick the posture that matches your actual risk tolerance and industry.
- Turn off training-data usage as a baseline: Make sure your organization understands and applies ChatGPT Enterprise data controls, and opt-out settings available on personal accounts, so prompts aren't used to train future models by default.
- Deploy detection, not just policy: A written acceptable-use policy is a start, but what you need is DLP for AI endpoints — a tool that can flag or block sensitive data, and CASB rules blocking pastes of regex-matched sensitive data before it ever leaves your network for a third-party AI tool.
- Give employees an approved alternative: This is the real lesson from the Amazon and Goldman case studies: banning a genuinely useful productivity tool without offering a sanctioned replacement doesn't eliminate the risk; it just moves it underground.
- Train for the "helpful assistant" blind spot: Awareness training needs to specifically address why competent, well-meaning employee use is the primary threat vector. Most people pasting sensitive data into ChatGPT genuinely believe they're just being efficient, and may not understand the very real risks of data breaches and more.
- Revisit the policy frequently: This space is moving fast with new models, enterprise controls, and extraction techniques still emerging. Treat your ChatGPT policy as a living document, not a memo you send once and forget.
Speaking of practical frameworks, check out our free guide to 12 Essential Cybersecurity Metrics
Why DLP tools are critical to managing ChatGPT risks
A dedicated DLP tool like Sonar doesn't just make data-sharing risk more visible after the fact; it can proactively flag or stop a risky paste into ChatGPT before it becomes an incident.
Combined with the seven-step framework above, this is one of the most effective ways to close the gap between simply having a policy in place and actually helping your teams make safer decisions.
With Sonar, you can protect even your least experienced or least security-aware users: the ones most likely to paste something sensitive without a second thought.
ChatGPT is a powerful tool — but it needs the right guardrails
Generative AI tools like ChatGPT are changing how we work, and for good reason. These tools can automate repetitive tasks and help teams get more out of their data and systems. But realizing those benefits safely means putting the right guardrails in place first, not after your own version of the Samsung or Amazon story makes headlines.
Try out our seven-step framework for yourself. And if you’d like to check out how a tool like Sonar can help your teams use generative AI tools like ChatGPT safely, then talk to one of our experts today.
FAQ
- Is ChatGPT safe to use at work? It can be, but you need the right controls in place. When used without policy, training-data opt-outs, or DLP monitoring, ChatGPT can create the real risk of confidential data leaving your organization's control.
- Can ChatGPT leak company data to other users? In theory, yes. Researchers have demonstrated that ChatGPT can be manipulated into outputting memorized fragments of its training data, including real personal information, through techniques like repeated-word prompting. OpenAI is continuing to patch specific known methods, but attackers are continuing to explore ways of getting ChatGPT to leak sensitive data.
- Does OpenAI train on my ChatGPT conversations? By default, consumer ChatGPT accounts may use conversation data to help improve future models, though users can opt out of this in settings. Enterprise and Team plans generally offer stronger default data controls, but organizations should confirm and configure these settings rather than assume they're already protected.
- Should companies ban ChatGPT? Not necessarily. The companies that navigated this best, like Goldman Sachs, didn't reject generative AI outright; they controlled how it was used, often while offering approved internal alternatives. A ban with no sanctioned alternative tends to push usage into shadow IT, where you lose visibility entirely.
