Generative AI Security Risks: What Compliance Teams Need to Know

Executive Summary

Most generative AI security risks are old security problems arriving through a new interface: data leaving the building, access that is too broad, suppliers nobody vetted. A few are genuinely new, namely prompt injection and the poisoning of data or models. A compliance team does not need a new control set for most of them. It needs to know which existing controls now apply to AI, and to write that down.

Generative AI security risks are the ways an AI system that creates text, code or images can be attacked, misused or made to leak data: prompt injection, sensitive data disclosure, poisoned data or models, supply chain weaknesses, insecure outputs, and agents given too much access. Each maps to controls you may already run.

By the end, you will know which risks are security risks, which framework names each one, which control you already run that covers it, and what evidence an auditor or enterprise buyer will ask to see.

Here's what I'll cover:

• Where each risk enters the system, with a table of examples
• Which framework does what: OWASP, NIST AI 600-1, MITRE ATLAS, ISO 42001 and the EU AI Act
• A crosswalk from each risk to ISO 27001, ISO 42001 and SOC 2
• Mitigation by layer, a five-step triage and a worked scenario
• The evidence to keep, a checklist and the mistakes to avoid

What Generative AI Security Risks Are and How They Differ From Ordinary Software Risk

Generative AI security risks are the vulnerabilities, attack paths and data exposures that come from systems built on large language models and similar generators, as opposed to the risks of ordinary deterministic software.

Three terms first. An LLM (large language model) is the text-predicting model underneath tools like ChatGPT. A prompt is the text sent to it. RAG (retrieval-augmented generation) is where the model looks things up in your documents before answering, and an agent is a model wired to tools it can call, such as a refund or deploy function.

These systems behave differently for three reasons. The model treats instructions and data as the same stream of text. Its outputs are not deterministic. And it is usually connected to internal data and tools, which is where the real damage happens.

Two Sides of Generative AI Security Risks: Attack Surface and Attacker Uplift

NIST's Generative AI Profile (NIST AI 600-1, July 2024) says generative AI systems present "two primary information security risks" in its Section 2.9. First, the technology can lower the barriers to offensive cyber capability such as hacking, malware and phishing. Second, it "expands the available attack surface, as GAI itself is vulnerable to attacks like prompt injection or data poisoning." The full text is in the NIST AI 600-1 PDF.

That gives a clean split: risks to your AI system, and risks from attackers who use AI. A compliance program mostly owns the first. The second is handled by the phishing, awareness and detection controls you already have.

How Generative AI Security Risks Differ From Traditional Application Risk

Why Generative AI Security Risks Land on Compliance Teams

The question is not whether generative AI is secure. It is whether you can show a customer or auditor how you control it.

That question arrives from three directions:

• Enterprise security questionnaires now carry AI sections: which models you use, what data they see, who approves their actions.
• SOC 2 and ISO 27001 scope changes when an LLM vendor holds customer data. The does SOC 2 cover AI guide walks through what auditors ask.
• The EU AI Act adds a legal duty for high-risk systems. Article 15 requires accuracy, robustness and cybersecurity, and its paragraph 5 names data poisoning, model poisoning, adversarial examples and confidentiality attacks. The Commission's AI Act Service Desk carries the text.

That last one does not apply to every generative feature. Article 15 covers high-risk systems, and under the Digital Omnibus the Annex III high-risk obligations now apply from 2 December 2027. Providers of general-purpose models with systemic risk have their own cybersecurity duty under Article 55. A support chatbot at a 30-person SaaS company is usually neither. Our EU AI Act compliance guide covers how to tell.

Generative AI Security Risks Examples: Where Each One Enters the System

The easiest way to see generative AI security risks examples is to follow one request through the system: input, retrieval, model, output, action. Each stage has its own failure, and OWASP has a name for most of them.

The OWASP column uses the entry titles from the OWASP Top 10 for LLM Applications. The NIST column is my mapping to the twelve risk names in AI 600-1, and where a risk straddles two names I chose the closer one.

Prompt Injection and Jailbreaks: The Risk Specific to Language Models

OWASP defines it this way: a prompt injection vulnerability "occurs when user prompts alter the LLM's behavior or output in unintended ways." Direct injection is a user typing the attack. Indirect injection is the model accepting input from external sources, such as websites or files, that carry hidden instructions.

Input filtering alone cannot close this, because the attack is ordinary language. OWASP lists seven mitigation areas: constrain model behavior, validate expected output formats, filter input and output, enforce least privilege, require human approval for high-risk actions, segregate and identify external content, and run adversarial testing.

Practitioners disagree on how solvable it is.

A Hacker News commenter called it "an unsolvable problem" in a July 2025 thread on a database-leak report. In a June 2026 thread, another commenter argued it is possible "to reject prompt injection to several nines" with different model designs, while conceding it is not solved today. For a compliance team the practical reading is the same either way: do not rely on the model to police itself, and limit what a fooled model can do.

Generative AI Data Privacy Risks: Leakage, Training Data and Retention

Generative AI data privacy risks reach you by three paths. Staff paste sensitive data into a tool. The model or its retrieval layer surfaces data the user should not see (OWASP LLM02 and LLM08). And prompts and outputs are logged and retained, by the vendor or by you.

NIST's Data Privacy risk covers the leakage, unauthorized use, disclosure or de-anonymization of personal or sensitive data. Retention is the path teams forget: a vendor that says it does not train on your data may still store your prompts for a set period.

Lawful basis, data protection impact assessments and automated-decision rules are a separate subject, handled in the GDPR-and-AI material rather than here.

Generative AI Security Risks in the Supply Chain: Poisoning and Retrieval Weaknesses

OWASP LLM03 (Supply Chain), LLM04 (Data and Model Poisoning) and LLM08 (Vector and Embedding Weaknesses) all describe the same exposure from different angles: you run things you did not build. NIST names it Value Chain and Component Integration, and describes third-party components such as procured datasets, pre-trained models and software libraries that "might be improperly obtained or not properly vetted."

Treat pre-trained models, datasets, plugins and connectors as suppliers. That means they belong in the same vendor review as any other supplier that touches customer data.

Generative AI Security Risks From Agents and Excessive Agency

OWASP LLM06 is about giving a model too much functionality, permission or autonomy. A read-only chatbot can leak what it can read. An agent with a refund tool or a deploy tool can also act on a bad instruction.

The two controls are old ones: least privilege, and a human approving high-impact actions. If an agent runs with the permissions of the developer who built it, that is the finding.

Generative AI Security Risks in Outputs: Improper Handling and Confabulation

OWASP LLM05 is model output passed to another system without validation. LLM09 is misinformation. NIST calls the broader phenomenon confabulation: generated content that is confidently presented but wrong.

The security angle is narrow and useful. Model output is untrusted input to whatever consumes it. Validate it before it reaches a browser, a database or a shell. Confabulation as a business and trust problem sits with whoever owns the risk assessment, not with security alone.

Shadow AI and Attacker Uplift: Two Risks Outside Your Own Build

Shadow AI is staff using AI tools nobody approved or inventoried. It is a data-exposure risk that your own build never touches, and it needs its own inventory and acceptable-use rule.

Attacker uplift is the other half of NIST's Information Security risk: easier phishing, malware and reconnaissance, and large-scale disinformation under Information Integrity. The control here is not a new AI control. It is the awareness training and detection you already run, such as ISO 27001 A.6.3.

Generative AI Security Framework Landscape: OWASP, NIST AI 600-1, MITRE ATLAS, ISO 42001 and the EU AI Act

No single framework is "the" generative AI security framework. They do different jobs, and confusing them is the commonest mistake in an AI questionnaire.

The MITRE description is quoted from the ATLAS data repository, which carries content version 2026.05.

The rule I give a small team is short, and it uses NIST AI 600-1 only for vocabulary. Use OWASP as the risk checklist, NIST AI 600-1 as the shared vocabulary, ISO 27001 or SOC 2 as the control base, and ISO 42001 if a customer asks for an AI-specific certification.

OWASP Top 10 for LLM Applications: The Ten Entries in Plain Language

The owner column is my suggestion, not part of OWASP's list. For the entries auditors raise most often, LLM02 and LLM06, the SOC 2 angle is in does SOC 2 cover AI.

NIST AI 600-1: The Twelve Risks and Which Ones Are Security Risks

The twelve risks in NIST AI 600-1 are CBRN Information or Capabilities, Confabulation, Dangerous, Violent or Hateful Content, Data Privacy, Environmental Impacts, Harmful Bias or Homogenization, Human-AI Configuration, Information Integrity, Information Security, Intellectual Property, Obscene, Degrading and/or Abusive Content, and Value Chain and Component Integration.

I count four as security risks this guide should cover: Information Security, Data Privacy, Value Chain and Component Integration, and Information Integrity. The others have different owners, and this guide does not try to teach them.

For the profile's place in the wider AI RMF, see the NIST AI RMF guide.

Generative AI Security Controls: Mapping Each Risk to ISO 27001, ISO 42001 and SOC 2

Most generative AI security controls already exist in an ISO 27001 or SOC 2 program. The work is applying them to AI systems and writing it down.

The crosswalk below pairs each risk with ISO/IEC 27001:2022 Annex A control titles, ISO/IEC 42001 Annex A control titles and the SOC 2 category it falls under. Titles only: the standards themselves are paywalled and I am not reproducing their text.

One honest caveat sits under this table. No Annex A control is named "prompt injection." Which control you attach a risk to is a judgment call, and you record it in your Statement of Applicability (the document that lists which controls apply and why). Two reasonable teams can map the same risk differently.

If you want the standards themselves explained, start with what is ISO 42001, the ISO 42001 checklist and ISO 27001 and AI compliance.

Generative AI Security Risks and Mitigation: What to Do for Each Risk

Generative AI security risks and mitigation go together, and the mitigation is layered. No single filter, policy or vendor stops these risks, which is why each one gets controls at more than one layer.

The first two policy documents are worth having in writing. Our acceptable use policy and AI governance policy pages cover both.

How to Assess Generative AI Security Risks Without a Security Team

You can triage in an afternoon. This is a screen, not a scoring method.

1. Inventory every AI feature and tool, including the ones staff adopted themselves.
2. Note what data each one touches, using your existing classification.
3. Note what actions each one can take: read only, write, send, spend, deploy.
4. Mark each against the OWASP ten and note which entries plausibly apply.
5. Pick the top three to fix first, favoring anything that both touches sensitive data and can act.

That stops short of likelihood and impact scoring on purpose. When you are ready for the register and the scoring, the AI risk assessment guide covers the method.

A Worked Scenario: One Support Copilot, Walked Through Risk by Risk

This scenario is illustrative, with no real company or numbers. A 30-person SaaS company adds a support copilot. It reads the help center, past tickets and customer accounts, and it can issue refunds up to a limit.

Walk it through the five stages. A customer ticket contains hidden instructions, which is indirect prompt injection. The copilot retrieves another customer's ticket because the index does not respect account permissions. The refund tool is called without approval. The reply is pasted into an internal system unvalidated. And the vendor retains the prompts.

None of the five failures needs a new kind of control. Each one needs an existing control applied to a system that did not exist last year, and a record that it works.

What Auditors and Customers Ask About Generative AI Security, and the Evidence to Keep

The questions about generative AI security controls are predictable. Each has an artifact that answers it, and if you cannot produce the artifact, the answer is "no" regardless of what is true in practice.

For SOC 2, these evidence items sit mostly under the Security and Confidentiality categories, with Privacy and Processing Integrity added where the system handles personal data or produces outputs customers rely on. For ISO, they map to the Annex A titles in the crosswalk above. The does SOC 2 cover AI guide has the auditor's side of this.

Important: ComplyJet does not test models or filter prompts. We keep the policies, risk records, vendor evidence and access evidence that an auditor reads, and the Trust Center is where you can show customers what you have in place.

Generative AI Security Checklist for Compliance Teams

This generative AI security checklist has ten items, grouped by what you are trying to do. Each ties back to a risk above.

Govern

1. Name one owner for AI security risk and publish an approved-tool list and acceptable use rule. (Data disclosure, shadow AI)
2. Add AI systems to the scope of your SOC 2 or ISO 27001 program, including vendors that hold customer data. (Supply chain)

Discover

3. Keep a current inventory of every AI feature and tool, with data class, ability to act, and owner. (All)
4. Record which vendors receive prompts, what they retain and whether they train on your data. (Data privacy)
5. Review the list quarterly and after any vendor model change. (Supply chain)

Protect

6. Give agents and connectors least-privilege access, never a developer's own credentials. (Excessive agency)
7. Require human approval for high-impact agent actions. (Excessive agency, prompt injection)
8. Validate model output before another system consumes it. (Improper output handling)
9. Make retrieval respect the permissions of the person asking. (Retrieval weaknesses)

Prove

10. Log prompts and tool calls, and keep adversarial test results and the incident procedure where an auditor can find them. (Every risk)

Mistakes That Leave Generative AI Security Risks Uncovered

These are easy to make, because most of them look like diligence.

• Treating the system prompt as a security boundary. A prompt that says "never reveal X" is a request, not a control.
• Scoping SOC 2 so AI vendors sit outside it. The report then says nothing about where customer data goes.
• Trusting "we do not train on your data" without reading retention terms. Not training is not the same as not storing.
• Giving agents the permissions of the developer who built them. Build a narrow role for the agent instead.
• Testing only the happy path. If nobody tried to break the feature, the test plan proves little.
• Mapping every risk to a new control instead of an existing one. It creates work nobody operates.
• Letting the AI inventory go stale, or treating confabulation as a security-only problem. The first hides new tools. The second sends a product and trust issue to the wrong team.

FAQs

What Are the Security Risks of Generative AI?

The main ones are prompt injection, sensitive data disclosure, poisoned data or models, supply chain weaknesses, insecure output handling and agents with too much access. Two more sit outside your own build: staff using unapproved tools, and attackers using AI to improve phishing and malware. NIST and OWASP both describe them, and the table in this guide shows where each one enters.

What Is Prompt Injection and Why Is It a Generative AI Security Risk?

Prompt injection is input that alters a model's behavior in ways you did not intend. It is a security risk because the model cannot reliably tell instructions from data, so text in a web page, file or ticket can steer it. Limit the damage with least privilege, human approval for high-risk actions and separation of external content.

How Can Employees Accidentally Expose Sensitive Data Through Generative AI?

The usual route is pasting customer data, source code or contracts into a tool the company has not approved. Prompts can also be logged and retained by the vendor, and a connected tool can surface documents the person should not see. An approved-tool list, data classification and a clear rule on what may be pasted address most of it.

Is ChatGPT Safe to Use for Business Data?

It depends on the plan, the contract and the data. Read the vendor's terms on retention, training use and subprocessors for the exact product your team uses, and classify what you allow into it. If the terms do not meet your customer commitments, restrict the data or use a different tool.

Which Frameworks Cover Generative AI Security Risks?

OWASP's Top 10 for LLM Applications is the risk checklist, and NIST AI 600-1 is the shared vocabulary. MITRE ATLAS describes adversary techniques, ISO 42001 is the certifiable AI management system, and ISO 27001 and SOC 2 provide the control base. EU AI Act Article 15 applies to high-risk systems only.

Does SOC 2 or ISO 27001 Cover Generative AI Security Risks?

Only to the extent that AI systems are inside the scope of the report or certificate. If an LLM vendor holds customer data and it is outside scope, the report says nothing about it. Existing controls such as access control, logging and supplier management do apply once you bring AI systems in.

How Do You Assess Generative AI Security Risk?

List every AI feature and tool, note the data each touches and the actions each can take, mark them against the OWASP ten, and fix the top three. That is a triage. For likelihood and impact scoring, use a formal AI risk assessment.

What Is the Difference Between Generative AI Security Risks and an AI Risk Assessment?

The security risks are the things that can go wrong from an attack or leak: injection, disclosure, poisoning. An AI risk assessment is the method that scores those and other risks, including fairness and reliability, and records decisions in a register. This guide names the risks, and the AI risk assessment guide scores them.

Is Generative AI a Compliance Risk?

It becomes one when it handles regulated data, falls under a law such as the EU AI Act, or is named in a customer contract or questionnaire. A tool used with no data, no customer exposure and no obligation is a smaller concern. Most SaaS teams are somewhere in between, which is why the inventory comes first.

Related Reading

• AI risk assessment: scoring the risks of one AI system and keeping the register
• Does SOC 2 cover AI?: where a SOC 2 report stops on AI
• NIST AI risk management framework: the framework the Generative AI Profile extends
• AI governance policy: approved-tool rules and a policy you can adapt
• ISO 27001 and AI compliance: bringing AI systems into an ISO 27001 scope
• What is ISO 42001: the certifiable AI management system
• ISO 42001 checklist: a step-by-step path to certification
• EU AI Act compliance guide: who Article 15 applies to, and when
• Acceptable use policy: the rule behind an approved-tool list