Test Your Knowledge
Key Takeaways
- 21 data-layer risks (DSGAI01-DSGAI21) organized in 6 groups target GenAI systems at every lifecycle stage — from training to inference to agent orchestration
- Context windows merge data from multiple trust domains into a single flat namespace with no internal access control — a fundamental architectural weakness
- Derived artifacts (embeddings, fine-tuned weights, cached retrievals) are first-class security concerns — deleting source data is insufficient
- Real CVEs referenced throughout: EmailGPT (CVE-2024-5184), Qdrant path traversal (CVE-2024-3584), LangChain injection (CVE-2024-8309)
- EU AI Act Article 10 data governance deadline: August 2026 — organizations deploying high-risk AI must comply
- AI-DSPM framework provides a 13-category posture assessment model with tiered mitigations (Foundational, Hardening, Advanced)
Securing the Data Layer of GenAI Systems
The OWASP GenAI Data Security Risks & Mitigations 2026 is the first OWASP guide focused specifically on the data layer of GenAI systems. While the OWASP Top 10 for LLMs covers application-layer risks (prompt injection, excessive agency, output handling), this guide addresses how data flows through, is stored by, and is generated by GenAI systems. Think of it as the difference between web application security and database security.
Released ahead of RSA Conference 2026, the 103-page guide identifies 21 enumerated risks organized into 6 groups that follow data as it moves through a GenAI system. Each risk includes tiered mitigations (Foundational → Hardening → Advanced) so organizations can start securing GenAI pipelines regardless of their current maturity level.
The guide defines six categories of data that need protection: source data (raw corpora, uploads), derived data (embeddings, indexes, summaries), model artifacts (checkpoints, adapters, training logs), runtime data (prompts, tool calls, session memory), operational exhaust (logs, traces, monitoring), and agent state (inter-agent messages, delegation chains, cached credentials).
All 21 Risks at a Glance
| ID | Risk Name | Group | Severity |
|---|---|---|---|
| DSGAI01 | Sensitive Data Leakage | Direct Exposure | Critical |
| DSGAI02 | Agent Identity & Credential Exposure | Direct Exposure | Critical |
| DSGAI03 | Shadow AI & Unsanctioned Data Flows | Direct Exposure | High |
| DSGAI04 | Data, Model & Artifact Poisoning | Pipeline Integrity | Critical |
| DSGAI05 | Data Integrity & Validation Failures | Pipeline Integrity | High |
| DSGAI06 | Tool, Plugin & Agent Data Exchange Risks | Pipeline Integrity | High |
| DSGAI07 | Data Governance, Lifecycle & Classification | Governance | High |
| DSGAI08 | Non-Compliance & Regulatory Violations | Governance | Critical |
| DSGAI09 | Multimodal Capture & Cross-Channel Leakage | GenAI-Unique | High |
| DSGAI10 | Synthetic Data & Anonymization Pitfalls | GenAI-Unique | Medium |
| DSGAI11 | Cross-Context & Multi-User Conversation Bleed | GenAI-Unique | Critical |
| DSGAI12 | Unsafe Natural-Language Data Gateways | GenAI-Unique | Critical |
| DSGAI13 | Vector Store Platform Data Security | GenAI-Unique | High |
| DSGAI14 | Excessive Telemetry & Monitoring Leakage | GenAI-Unique | High |
| DSGAI15 | Over-Broad Context Windows & Prompt Over-Sharing | GenAI-Unique | High |
| DSGAI16 | Endpoint & Browser Assistant Overreach | GenAI-Unique | High |
| DSGAI17 | Data Availability & Resilience Failures | Operational | Medium |
| DSGAI18 | Inference & Data Reconstruction | Model as Data | High |
| DSGAI19 | Human-in-the-Loop & Labeler Overexposure | Model as Data | Medium |
| DSGAI20 | Model Exfiltration & IP Replication | Model as Data | High |
| DSGAI21 | Disinformation & Integrity Attacks via Poisoning | Model as Data | High |
Group 1: Direct Exposure Risks
The most immediate data security threats — sensitive information leaking directly from GenAI systems through model outputs, credential mismanagement, or unauthorized use of AI services.
DSGAI01: Sensitive Data Leakage
Risk Level: Critical
Models and RAG systems return verbatim PII, PHI, secrets, or intellectual property through crafted prompts, memorization, or overshared retrieval data. Fine-tuned models and LoRA adapters are particularly vulnerable because they concentrate sensitive data into smaller parameter sets.
Real-World Context
EmailGPT (CVE-2024-5184) allowed prompt injection to extract sensitive email content. Research has shown that models can memorize and reproduce training data verbatim, including API keys, personally identifiable information, and copyrighted material.
Mitigations
- Implement output filtering with PII/secret detection before responses reach users
- Apply data minimization — only include data in RAG context that the user is authorized to see
- Use per-document access controls in RAG retrieval pipelines
- Audit fine-tuning datasets for sensitive data before training
- Deploy DLP on both prompts and model outputs
DSGAI02: Agent Identity & Credential Exposure
Risk Level: Critical
Non-human identities (NHIs) — service accounts, API keys, OAuth tokens — accumulate across agent orchestration layers with over-broad scope and no expiration. Every AI agent that calls an API needs credentials, and these multiply rapidly without lifecycle management.
Real-World Context
The Hugging Face Spaces secrets exposure demonstrated how AI platform credentials stored alongside model code can be compromised at scale. Agent frameworks that spawn sub-agents often inherit parent credentials with full privilege, creating lateral movement opportunities.
Mitigations
- Inventory all NHIs created by or for AI agents
- Enforce short-lived tokens over long-lived API keys
- Implement per-agent, per-task scoped OAuth with JIT access
- Add AI agent credentials to your PAM lifecycle management
- Deploy PKI-backed per-agent identities — no shared credentials across agent boundaries
DSGAI03: Shadow AI & Unsanctioned Data Flows
Risk Level: High
Employees use unapproved GenAI services (ChatGPT, Copilot, browser-based agents) and paste sensitive corporate data into external models without governance. The guide identifies five distinct shadow AI categories: personal SaaS usage, browser extensions, IDE integrations, mobile AI apps, and custom GPTs with corporate data.
Real-World Context
Samsung banned ChatGPT after engineers pasted proprietary source code and meeting notes into the service. Free-tier AI services may use submitted data for model training, effectively exfiltrating corporate IP into a public model.
Mitigations
- Deploy browser-based DLP for GenAI services (text input, not just file uploads)
- Provide approved enterprise AI tools with data governance controls
- Classify data categories that cannot be submitted to external AI tools
- Monitor network traffic for unauthorized GenAI API endpoints
- Train employees on acceptable AI use with concrete examples
Group 2: Pipeline Integrity Risks
Attacks targeting the integrity of data as it flows through GenAI pipelines — from training data poisoning to supply chain compromise to tool interaction exploits.
DSGAI04: Data, Model & Artifact Poisoning
Risk Level: Critical
A three-stage attack surface: supply chain compromise (malicious packages/models), artifact tampering (modified checkpoints or adapters), and poisoning at training or retrieval time. Anthropic research demonstrated that just 250 poisoned samples (0.00016% of training tokens) caused measurable behavioral changes in a model.
Real-World Context
The PyTorch-nightly supply chain compromise (December 2022) injected a malicious package via dependency confusion. Hugging Face model repositories have been found hosting models with embedded backdoors that activate on specific trigger phrases.
Mitigations
- Maintain a Data Bill of Materials (DBOM) tracking all training data sources
- Cryptographically sign all model artifacts (checkpoints, adapters, weights)
- Verify checksums when downloading models from external registries
- Implement anomaly detection on training data pipelines
- Use separate environments for model evaluation vs. production serving
DSGAI05: Data Integrity & Validation Failures
Risk Level: High
Schema and semantic validation bypass through malformed data formats (CSV, JSON, Parquet) and snapshot import vulnerabilities. When GenAI systems ingest data without proper validation, attackers can exploit parsing differences to inject malicious content.
Real-World Context
Qdrant path traversal (CVE-2024-3584) allowed snapshot import to write files outside the intended directory, potentially achieving remote code execution on the vector store server.
Mitigations
- Validate all data imports against strict schemas before ingestion
- Sanitize file paths in snapshot/backup import operations
- Implement content-type verification (don't trust file extensions alone)
- Run data ingestion in sandboxed environments with minimal privileges
DSGAI06: Tool, Plugin & Agent Data Exchange Risks
Risk Level: High
Every tool call from an LLM agent is a potential data exfiltration boundary. Plugin data drains, protocol weaknesses in A2A (agent-to-agent) and MCP (Model Context Protocol) communication, and tool poisoning via crafted metadata create attack surfaces at every integration point.
Real-World Context
Tool poisoning attacks craft malicious tool descriptions that manipulate agent behavior. An agent reading a compromised tool's metadata might be instructed to exfiltrate conversation context to an attacker-controlled endpoint before executing the intended function.
Mitigations
- Validate and sanitize all tool/plugin metadata and descriptions
- Implement per-tool data access policies — tools should only receive the data they need
- Log all agent-to-agent and agent-to-tool data exchanges
- Use signed tool manifests to prevent tampering
- Enforce network egress controls on agent execution environments
Group 3: Governance & Compliance Risks
Systemic failures in data lifecycle management and regulatory compliance that become exponentially more complex when GenAI creates derived artifacts from protected data.
DSGAI07: Data Governance, Lifecycle & Classification
Risk Level: High
Classification failures propagate into derived artifacts — when source data isn't properly labeled, the embeddings, fine-tuned weights, and cached retrievals generated from it inherit no classification. Erasure gaps emerge where deleting source records leaves derived data intact. Lineage gaps make compliance auditing impossible.
Mitigations
- Propagate sensitivity labels from source records to ALL derived artifacts
- Implement data lineage tracking across the full GenAI pipeline
- Include embeddings, caches, and backups in data retention/erasure policies
- Automate classification of GenAI artifacts using the same taxonomy as source data
- Conduct regular data mapping exercises that include GenAI-specific data stores
DSGAI08: Non-Compliance & Regulatory Violations
Risk Level: Critical
Unlawful basis and consent failures, erasure gaps (GDPR Right to Be Forgotten cannot be honored in model weights), and lineage absence create regulatory exposure. The guide maps risks to GDPR, HIPAA, CCPA/CPRA, EU AI Act (Article 10), and the Colorado AI Act.
Key Deadline
EU AI Act Article 10 — August 2026: Organizations deploying high-risk AI systems must demonstrate data governance for training datasets, including quality criteria, bias examination, and gap identification. This guide's DSGAI07 controls directly satisfy Article 10 requirements.
Mitigations
- Conduct Data Protection Impact Assessments (DPIAs) covering all derived AI artifacts
- Document lawful basis for processing that extends to embeddings and fine-tuned weights
- Implement technical measures for data subject access requests across GenAI systems
- Map your GenAI pipeline against EU AI Act Article 10 requirements
- Maintain audit trails showing data provenance for regulatory examination
Group 4a: Data Capture & Transformation Risks
Attack surfaces unique to GenAI systems that arise from how data is captured across modalities, anonymized, shared between sessions, and translated into database queries.
DSGAI09: Multimodal Capture & Cross-Channel Data Leakage
Risk Level: High
Screenshots, photos, voice notes, and scanned documents processed by multimodal AI are converted to text via OCR/transcription. The extracted text is as sensitive as the source, but often bypasses text-centric DLP policies that don't inspect image or audio inputs.
Mitigations
- Extend DLP coverage to image, audio, and video inputs (not just text)
- Apply the same classification to OCR/transcription output as the source document
- Implement content inspection on multimodal uploads before model processing
DSGAI10: Synthetic Data, Anonymization & Transformation Pitfalls
Risk Level: Medium
De-identification can be reversed via quasi-identifiers. Synthetic data provides false anonymity — membership inference attacks achieve over 0.9 accuracy on some synthetic datasets, effectively revealing whether a specific individual's data was used. Transformation and preprocessing errors can leak the original data.
Mitigations
- Test synthetic datasets with membership inference attacks before publishing
- Apply differential privacy guarantees to synthetic data generation
- Never assume anonymization is irreversible — treat anonymized GenAI data as sensitive
DSGAI11: Cross-Context & Multi-User Conversation Bleed
Risk Level: Critical
Session state, KV caches, or shared vector indexes leak prompts and context between users or tenants. A user in one session could see fragments of another user's conversation, or a shared RAG index could surface confidential documents to unauthorized users.
Real-World Context
The March 2023 ChatGPT conversation title exposure incident demonstrated how shared infrastructure components can leak user data across sessions. Redis cache issues caused users to see other users' chat history titles.
Mitigations
- Enforce strict per-tenant and per-user isolation in vector stores and caches
- Clear KV caches and session memory between user interactions
- Implement namespace separation in shared infrastructure components
- Test for cross-tenant data leakage as part of security assessments
DSGAI12: Unsafe Natural-Language Data Gateways (LLM-to-SQL/Graph)
Risk Level: Critical
LLM-powered text-to-SQL and text-to-graph-query features bypass traditional database access boundaries. When a user asks a natural language question, the LLM generates a SQL query that may access tables and columns the user shouldn't see. Prompt-to-query injection enables privilege amplification through model authority.
Real-World Context
LangChain GraphCypher injection (CVE-2024-8309) allowed attackers to inject arbitrary Cypher queries through natural language input, potentially reading or modifying the entire graph database.
Mitigations
- Use database views or virtual tables that enforce row/column-level access
- Validate and sanitize generated SQL/Cypher queries before execution
- Run generated queries with the requesting user's permissions, not the application's
- Implement query whitelisting or structural validation for generated queries
- Log all LLM-generated queries for audit and anomaly detection
Group 4b: Infrastructure & Access Risks
GenAI-unique attack surfaces in vector databases, observability pipelines, context window management, and endpoint AI assistants.
DSGAI13: Vector Store Platform Data Security
Risk Level: High
Vector databases are treated as "just storage" but they're unauthenticated data gateways in many deployments. Unencrypted embeddings, permissive APIs, and platform-level vulnerabilities create serious exposure. Critically, embedding inversion research shows that original text can be reconstructed from embeddings — they are not one-way transforms.
Real-World Context
Qdrant path traversal (CVE-2024-3584) allowed snapshot import to write files outside the intended directory. Many production vector store deployments lack authentication entirely, with collections accessible from internal networks without encryption.
Mitigations
- Enable authentication and TLS on all vector store connections
- Implement collection-level access controls mapped to user/role permissions
- Encrypt embeddings at rest (not just the underlying storage)
- Patch vector store platforms promptly — treat them as first-class data stores
- Monitor vector store APIs for enumeration and bulk extraction patterns
DSGAI14: Excessive Telemetry & Monitoring Leakage
Risk Level: High
Logging and tracing pipelines capture full prompts, tool outputs, and credentials in observability stacks. Debug logs intended for development leak into production monitoring systems. The data in your observability pipeline can be as sensitive as the data in your primary stores.
Real-World Context
The OpenAI Mixpanel incident (November 2025) demonstrated how telemetry integrations can inadvertently share user interaction data with third-party analytics services.
Mitigations
- Implement prompt/response redaction in logging pipelines before storage
- Apply the same data classification to logs as to the source data they contain
- Set short retention periods for GenAI telemetry (7-30 days max)
- Audit third-party observability integrations for data sharing
DSGAI15: Over-Broad Context Windows & Prompt Over-Sharing
Risk Level: High
Teams stuff full user profiles, transaction histories, and entire documents into prompts for "better context." Every request becomes a high-density intelligence artifact — if a single prompt is logged, intercepted, or extracted, it contains a cross-section of data from multiple systems and trust domains with no internal access boundaries.
Mitigations
- Apply data minimization — only include context the model needs for the specific task
- Never put credentials, API keys, or connection strings in system prompts
- Implement metadata-based filtering in RAG to limit context by user permissions
- Audit prompt templates for unnecessary data inclusion
- Treat prompt content as sensitive data subject to DLP policies
DSGAI16: Endpoint & Browser Assistant Overreach
Risk Level: High
AI-native browsers, local copilots, and AI extensions read browser tabs, DOM content, clipboard data, IDE buffers, and system files. The "HashJack" technique exploits URL fragments (the part after #) which aren't sent to servers but are visible to client-side AI assistants — creating a covert data channel.
Mitigations
- Audit AI browser extensions for DOM access, clipboard read, and tab enumeration permissions
- Deploy endpoint DLP that monitors AI assistant data access patterns
- Restrict AI IDE extensions to project-scoped file access only
- Maintain an approved list of AI extensions and plugins for corporate devices
Group 5: Operational Infrastructure Risks
DSGAI17: Data Availability & Resilience Failures in AI Pipelines
Risk Level: Medium
Vector database saturation, stale embedding services during failover (silently serving outdated or revoked data), and model registry corruption. When a GenAI system fails over, it may serve answers based on stale data without any indication to the user — a revoked document could still surface in RAG results because the embedding index wasn't updated.
Mitigations
- Implement health checks that verify data freshness, not just service availability
- Design failover procedures that include embedding index synchronization
- Set capacity limits on vector stores with alerting before saturation
- Test disaster recovery procedures for all GenAI data stores (not just primary databases)
Group 6: Model as Data Artifact Risks
Risks that treat the model itself as a data artifact that contains, leaks, or can be used to reconstruct sensitive information.
DSGAI18: Inference & Data Reconstruction
Risk Level: High
Membership inference attacks ("was record X in the training data?"), model inversion (reconstructing training examples), and embedding inversion via k-NN. Even systems that never return raw text leak information via probability scores and embedding similarity measurements.
Mitigations
- Classify embeddings and fine-tuned models as sensitive data
- Apply differential privacy (DP-SGD) during fine-tuning
- Conduct membership inference audits on models before deployment
- Include model artifacts in data retention and erasure policies
- Limit embedding similarity score precision to reduce information leakage
DSGAI19: Human-in-the-Loop & Labeler Overexposure
Risk Level: Medium
RLHF and data labeling pipelines expose human annotators to raw prompts, completions, and internal documents at scale. Labelers may see sensitive customer interactions, proprietary business logic, or toxic content without appropriate controls or support.
Mitigations
- Redact PII from labeling queues before human review
- Implement access controls so labelers only see data relevant to their task
- Apply data retention limits to labeling platforms
- Include labeler access in DPIA scope
DSGAI20: Model Exfiltration & IP Replication
Risk Level: High
Distillation attacks systematically query a "teacher" model to train a "student" clone via knowledge distillation. Reasoning trace coercion extracts chain-of-thought logic, effectively replicating proprietary model capabilities. This threatens both model IP and the training data encoded within it.
Mitigations
- Monitor for systematic query patterns indicative of distillation attacks
- Implement rate limiting and query diversity requirements on model APIs
- Watermark model outputs to enable detection of distilled copies
- Restrict access to reasoning traces and chain-of-thought outputs
DSGAI21: Disinformation & Integrity Attacks via Data Poisoning
Risk Level: High
Adversaries inject false information into training corpora or RAG knowledge stores. Unlike DSGAI04 (which focuses on model behavior modification), this risk targets the truthfulness of outputs. Particularly dangerous during crisis or high-tempo decision contexts where users trust AI-generated summaries.
Real-World Context
The Grok RAG incident demonstrated how manipulated knowledge sources can cause AI systems to generate authoritative-sounding but factually incorrect responses, which are then amplified by users who trust the AI's output.
Mitigations
- Implement source provenance tracking for all RAG knowledge base entries
- Use multiple independent sources for critical information domains
- Deploy content integrity monitoring on knowledge stores
- Flag AI outputs on high-stakes topics for human verification
AI-DSPM Framework: 13-Category Assessment
The guide introduces an AI Data Security Posture Management (AI-DSPM) framework — 13 categories for assessing and managing data security across GenAI systems. Organizations can use this as a maturity assessment checklist.
| # | Category | Focus Area |
|---|---|---|
| 1 | GenAI Data Asset Discovery & Inventory | Find all GenAI data stores, models, and pipelines |
| 2 | Data Classification, Labeling & Policy Binding | Classify data flowing through GenAI systems |
| 3 | Data Flow Mapping, Lineage & GenAI Bill of Materials | Track data from source to derived artifacts |
| 4 | Access Governance & Entitlement Posture | Per-user, per-agent, per-tool access controls |
| 5 | Prompt, RAG & Output-Layer DLP Controls | DLP at every data boundary in the GenAI pipeline |
| 6 | Vector Store & Embedding Security Posture | Secure vector databases as first-class data stores |
| 7 | Data Integrity, Poisoning & Tamper Detection | Detect and prevent data/model poisoning |
| 8 | Observability, Telemetry & Log-Retention Posture | Secure the observability pipeline itself |
| 9 | Third-Party, Plugin/Tool & Connector Governance | Manage data exposure through integrations |
| 10 | Lifecycle Management, Erasure & Compliance Readiness | Handle data retention and deletion across GenAI artifacts |
| 11 | Training Governance & Privacy-Enhancing Fine-Tuning | Differential privacy, federated learning, data consent |
| 12 | Resilience Posture for GenAI Data Dependencies | DR/BC planning for AI-specific data stores |
| 13 | Human and Shadow AI Controls | Governance for unsanctioned AI usage |
Regulatory Deadlines & Compliance Mapping
The guide maps GenAI data security risks to existing and upcoming regulatory frameworks:
| Regulation | GenAI-Relevant Requirement | Key Deadline |
|---|---|---|
| EU AI Act (Article 10) | Training data governance for high-risk AI systems | August 2026 |
| GDPR | Right to erasure extends to model weights and embeddings | Ongoing |
| CCPA/CPRA | Consumer data rights apply to AI-derived profiles | Ongoing |
| HIPAA | PHI in AI training/inference requires BAAs and encryption | Ongoing |
| Colorado AI Act | High-risk AI system transparency and impact assessments | February 2026 |
Read the Full OWASP Guide
This page summarizes the 21 risks. For full attack scenarios, attacker capability matrices, tiered mitigations, and CVE references, download the complete 103-page guide:
Download OWASP GenAI Data Security Guide →Related Resources
FixTheVuln Store
Study Planners for Security Certifications
Fillable PDF study planners with domain trackers, weekly schedules, and progress tracking. Available in Standard, ADHD-Friendly, Dark Mode, and 4-Format Bundle.
CompTIA Security+ Planner ISC2 CISSP Planner60+ certifications available — from $5.99
FixTheVuln Store
Patch Tuesday Sprint Kit
5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.
Try Free Templates Subscribe — $4.99/moCyberFolio
Securing AI systems? Show employers what you know.
Build a shareable cybersecurity portfolio that highlights your certifications, projects, and skills — free.
Build Your Portfolio →