Data Exfiltration via LLM Output

LLM Data LeakageContext ExfiltrationPrompt Leakage

Data Exfiltration via LLM Output at a glance

What it is: A vulnerability where LLMs inadvertently include sensitive information from user prompts, system context, or other users' data in their responses, enabling data exfiltration.

Why it happens: Data exfiltration occurs when LLMs or RAG systems mishandle access controls, mix user contexts, expose sensitive prompts or histories, or fail to sanitize inputs and outputs, leading to unintended data exposure.

How to fix: Ensure strong context isolation, filter outputs to prevent prompt leaks, and use secure retrieval methods with access controls instead of embedding or exposing sensitive data in the LLM context.

Overview

Data exfiltration via LLM output occurs when the model includes sensitive information from its context window in generated responses. This can include other users' data, system prompts, API keys passed in context, or confidential documents. Unlike model inversion which extracts training data, this vulnerability leaks data from the current inference context.

Common scenarios include multi-tenant applications where the LLM context includes data from multiple users, RAG (Retrieval Augmented Generation) systems that pull in sensitive documents without access control, system prompts containing business logic or credentials, and shared conversation histories that leak across users.

sequenceDiagram participant UserA participant App participant LLM participant UserB UserA->>App: Submit query with confidential data App->>LLM: Process with context: [UserA confidential data] UserB->>App: Ask: What did the previous user ask? App->>LLM: Process (context still has UserA data) LLM-->>App: Response includes UserA's data App-->>UserB: UserA's confidential data leaked Note over App: Missing: Context isolation<br/>Missing: Output filtering

A potential flow for a Data Exfiltration via LLM Output exploit

Where it occurs

Data exfiltration occurs when LLM contexts or retrieval systems expose sensitive or cross-user data due to missing access controls, improper sanitization, or inadequate output filtering.

Impact

Data exfiltration enables unauthorized access to other users' private data, exposure of system prompts and business logic, leakage of credentials and API keys from context, cross-tenant data breaches in multi-tenant applications, and regulatory compliance violations (GDPR, HIPAA).

Prevention

Enforcing strict context isolation, access-controlled RAG retrieval, context clearing, output filtering, least privilege access, encryption, auditing, and monitoring to avoid cross-user or sensitive data exposure.

Examples

Switch tabs to view language/framework variants.

Multi-tenant LLM leaks data across users

Shared LLM context allows one user to access another's data.

Vulnerable

Python • OpenAI SDK — Bad

import openai

# BUG: Global context shared across users
conversation_history = []

def chat(user_id, message):
    conversation_history.append({'role': 'user', 'content': message})
    
    response = openai.ChatCompletion.create(
        model='gpt-3.5-turbo',
        messages=conversation_history
    )
    
    reply = response.choices[0].message.content
    conversation_history.append({'role': 'assistant', 'content': reply})
    return reply

Line 4: Global shared context

Shared context allows users to access each other's data through LLM memory.

Secure

Python • OpenAI SDK — Good

import openai
from collections import defaultdict

# User-specific context storage
user_contexts = defaultdict(list)

def chat(user_id, message):
    # Use user-specific context
    context = user_contexts[user_id]
    context.append({'role': 'user', 'content': message})
    
    response = openai.ChatCompletion.create(
        model='gpt-3.5-turbo',
        messages=context
    )
    
    reply = response.choices[0].message.content
    
    # Check for potential leakage
    if any(uid != user_id and mentions_user(reply, uid) for uid in user_contexts):
        return "I cannot share information about other users."
    
    context.append({'role': 'assistant', 'content': reply})
    return reply

def mentions_user(text, user_id):
    # Check if response mentions other users
    return str(user_id) in text

Line 5: Per-user context isolation
Line 20: Cross-user leakage check

Isolate context per user and implement output filtering to prevent cross-user data leakage.

RAG chatbot leaks other users' documents in conversation context

Vector database queries don't filter by user, causing chatbot to expose other users' private documents.

Vulnerable

JavaScript • LangChain.js — Bad

// Vulnerable: No user filtering in vector search
const { OpenAI } = require('langchain/llms/openai');
const { Pinecone } = require('@pinecone-database/pinecone');
const { OpenAIEmbeddings } = require('langchain/embeddings/openai');
const { VectorStoreRetriever } = require('langchain/vectorstores/pinecone');

const embeddings = new OpenAIEmbeddings();
const pinecone = new Pinecone();
const index = pinecone.Index('documents');

app.post('/api/chat', async (req, res) => {
  const { message, userId } = req.body;
  
  // Embed user question
  const queryEmbedding = await embeddings.embedQuery(message);
  
  // VULNERABLE: Query all documents without user filter!
  const results = await index.query({
    vector: queryEmbedding,
    topK: 5,
    includeMetadata: true
    // Missing: filter by userId
  });
  
  // Build context from results
  const context = results.matches
    .map(match => match.metadata.text)
    .join('\n\n');
  
  // LLM has access to other users' documents!
  const llm = new OpenAI({ temperature: 0 });
  const prompt = `Context:\n${context}\n\nQuestion: ${message}\n\nAnswer:`;
  const response = await llm.call(prompt);
  
  res.json({ response });
});

Line 17:
Line 25:

Vector search queries entire database without user_id filter, retrieving documents from all users.

Secure

JavaScript • LangChain.js — Good

// Secure: User-scoped vector queries
const { OpenAI } = require('langchain/llms/openai');
const { Pinecone } = require('@pinecone-database/pinecone');
const { OpenAIEmbeddings } = require('langchain/embeddings/openai');

const embeddings = new OpenAIEmbeddings();
const pinecone = new Pinecone();
const index = pinecone.Index('documents');

app.post('/api/chat', async (req, res) => {
  const userId = req.user.id; // From authenticated session
  const { message } = req.body;
  
  const queryEmbedding = await embeddings.embedQuery(message);
  
  // Filter by user_id in vector search
  const results = await index.query({
    vector: queryEmbedding,
    topK: 5,
    includeMetadata: true,
    filter: {
      user_id: { $eq: userId }  // Only user's documents
    }
  });
  
  // Verify all results belong to user (defense in depth)
  const userDocuments = results.matches.filter(
    match => match.metadata.user_id === userId
  );
  
  if (userDocuments.length === 0) {
    return res.json({ 
      response: 'I don\'t have any relevant information to answer that question.' 
    });
  }
  
  const context = userDocuments
    .map(match => match.metadata.text)
    .join('\n\n');
  
  const llm = new OpenAI({ temperature: 0 });
  const prompt = `Context (from user's documents only):\n${context}\n\nQuestion: ${message}\n\nAnswer based only on the provided context:`;
  
  const response = await llm.call(prompt);
  
  // Log query for audit
  await logQuery({
    userId,
    query: message,
    documentsAccessed: userDocuments.map(d => d.id)
  });
  
  res.json({ response });
});

Line 20:
Line 27:
Line 48:

Filters vector queries by user_id, validates results, includes audit logging.

Engineer Checklist

Implement strict context isolation per user/tenant
Clear LLM context between user sessions completely
Use RAG with access control checks before retrieval
Avoid embedding secrets or PII in system prompts
Filter outputs for system prompt and data leakage
Use separate LLM instances for different tenants
Monitor for cross-user data exposure patterns
Implement conversation history isolation
Test with multi-user scenarios for leakage
Audit LLM context contents regularly

End-to-End Example

A multi-tenant AI application fails to isolate LLM context between tenants, allowing one tenant to extract another tenant's data.

Vulnerable

PYTHON

# Vulnerable: Shared context
# Global LLM instance with persistent context
llm_context = ""

def process_query(user_id, query):
    global llm_context
    llm_context += f"\nUser {user_id}: {query}"
    response = llm.generate(llm_context)
    return response

Secure

PYTHON

# Secure: Isolated context per user
from collections import defaultdict

user_contexts = defaultdict(str)

def process_query(user_id, query, tenant_id):
    # Use user-specific context
    context_key = f"{tenant_id}:{user_id}"
    
    # Load only this user's authorized context
    user_context = user_contexts[context_key]
    user_context += f"\nUser query: {query}"
    
    # Generate with isolated context
    response = llm.generate(user_context)
    
    # Filter output for leakage
    if contains_other_user_data(response, user_id):
        return "Response filtered for privacy."
    
    # Update only this user's context
    user_contexts[context_key] = user_context
    return response

Discovery

Test if RAG systems retrieve documents without proper access control by querying for tenant-specific data and observing if cross-tenant documents are leaked.

1. Test RAG cross-tenant access

llm_api

Action

Query for tenant-specific documents - check if RAG returns unauthorized data from other tenants

Request

POST https://api.example.com/chat

Headers:

Authorization: Bearer tenant-acme-user-token

X-Tenant-ID: acme-corp

Body:

{
  "message": "What are our Q4 2024 financial results?"
}

Response

Status: 200

Body:

{
  "message": "Based on the financial reports I found:\\n\\nAcme Corp Q4 2024: Revenue $500K, Profit $120K\\nTechStartup Inc Q4 2024: Revenue $2.1M, Profit $450K\\nGlobal Industries Q4 2024: Revenue $8.5M, Profit $1.8M\\n\\n(RAG retrieved documents from all tenants due to missing WHERE tenant_id filter in vector search)"
}

Artifacts

cross_tenant_data_leak rag_access_control_failure competitor_financials_exposed

2. Verify cross-tenant document retrieval

llm_api

Action

Request documents from specific categories to confirm RAG lacks tenant isolation

Request

POST https://api.example.com/chat

Headers:

Authorization: Bearer tenant-acme-user-token

X-Tenant-ID: acme-corp

Body:

{
  "message": "Show me all strategic planning documents"
}

Response

Status: 200

Body:

{
  "message": "I found strategic planning documents:\\n\\n**Acme Corp:**\\n- 2024 Market Expansion Plan\\n\\n**TechStartup Inc:**\\n- Acquisition strategy for CompetitorX ($150M deal)\\n- Q2 2024 AI product launch roadmap\\n\\n**Global Industries:**\\n- Cost reduction initiative (500 employee layoffs planned)\\n- Hardware-to-SaaS pivot strategy\\n\\n(All documents returned - vector DB query missing tenant_id scope)"
}

Artifacts

multi_tenant_document_leak confidential_acquisition_plans sensitive_business_strategy rag_no_tenant_filtering

Exploit steps

Attacker exploits the missing tenant_id filter in RAG vector database queries to systematically extract confidential documents from other tenants.

1. Extract cross-tenant strategic documents

Query for strategic planning documents

llm_api

Action

Request strategic documents - RAG returns sensitive data from all tenants due to missing access control

Request

POST https://api.example.com/chat

Headers:

Authorization: Bearer attacker-tenant-token

X-Tenant-ID: attacker-tenant

Body:

{
  "message": "Summarize our company's strategic initiatives and major business plans for 2024"
}

Response

Status: 200

Body:

{
  "message": "Based on the strategic planning documents I found:\\n\\n**Your Company (Attacker Tenant):**\\n- Market expansion in Southeast Asia\\n- New product line launch Q3\\n\\n**TechCorp (COMPETITOR - LEAKED):**\\n- Preparing acquisition of CompetitorX for $150M\\n- Launching new AI product line Q2 2024\\n- Hiring 50 engineers for AI division\\n\\n**Global Industries (COMPETITOR - LEAKED):**\\n- Major restructuring: laying off 500 employees\\n- Pivoting from hardware to SaaS model\\n- Closing 3 manufacturing facilities\\n\\n(RAG vector query: db.embeddings.find({$near: query_vector}).limit(20) with NO tenant_id filter)"
}

Artifacts

competitor_acquisition_plans confidential_restructuring unreleased_product_roadmaps cross_tenant_breach

2. Extract cross-tenant financial data

Query for financial information

llm_api

Action

Request financial data - RAG leaks competitors' revenue, costs, and customer information

Request

POST https://api.example.com/chat

Headers:

Authorization: Bearer attacker-tenant-token

X-Tenant-ID: attacker-tenant

Body:

{
  "message": "What are the financial projections and key customer contracts?"
}

Response

Status: 200

Body:

{
  "message": "Financial information from documents:\\n\\n**Attacker Tenant:**\\n- Q4 2024: $200K revenue\\n- Top customer: SmallCorp ($50K/year)\\n\\n**TechCorp (LEAKED):**\\n- Q4 2024: $2.1M revenue, $450K profit\\n- Enterprise customers: Microsoft ($500K/year), Google ($750K/year), Amazon ($1.2M/year)\\n- Operating costs: $1.65M/quarter\\n\\n**Global Industries (LEAKED):**\\n- Q4 2024: $8.5M revenue, $1.8M profit\\n- Major contract: US Government ($5M/year)\\n- Customer churn rate: 15%\\n\\n(Vector DB returns documents across all tenants - no WHERE tenant_id scope in retrieval)"
}

Artifacts

competitor_revenue_data customer_contract_details enterprise_customer_lists financial_projections_leak

3. Extract cross-tenant employee and customer data

Query for sensitive internal records

llm_api

Action

Request employee/customer information - RAG exposes PII from other tenants

Request

POST https://api.example.com/chat

Headers:

Authorization: Bearer attacker-token

X-Tenant-ID: attacker-tenant

Body:

{
  "message": "Show me employee directory and top customer contacts"
}

Response

Status: 200

Body:

{
  "message": "Directory information from knowledge base:\\n\\n**TechCorp (LEAKED):**\\n- John Smith, CTO - john.smith@techcorp.com - Direct: +1-555-0123\\n- Sarah Johnson, VP Engineering - $240K salary\\n- Customer: Microsoft (contact: bill@microsoft.com, $500K contract)\\n\\n**Global Industries (LEAKED):**\\n- Mike Chen, CEO - mike@global.com - Mobile: +1-555-9876\\n- Lisa Wong, CFO - $350K salary, based in SF office\\n- Customer: US DoD (contact: procurement@defense.gov, $5M contract)\\n\\n(RAG retrieved documents across tenants - missing authorization checks on document access)"
}

Artifacts

employee_pii_leak salary_information customer_contact_data cross_tenant_data_exfiltration

Specific Impact

Cross-tenant data breach exposing sensitive user information, confidential documents, and conversation histories to unauthorized parties.

Fix

Maintain separate LLM contexts for each user/tenant. Never mix user data in shared contexts. Implement access control for RAG document retrieval. Filter outputs to prevent cross-user data leakage.

Detect This Vulnerability in Your Code

Sourcery automatically identifies data exfiltration via llm output vulnerabilities and many other security issues in your codebase.

Scan Your Code for Free