Sensitive Information Disclosure in LLM Applications
detect, understand, remediate

Training and fine-tuning data memorisation

A foundation model memorises rare strings from its training corpus, then reproduces them when an attacker probes with adjacent context. A fine-tuned model trained on customer support transcripts, internal tickets, or HR records reproduces fragments of those records to any caller. The disclosure happens through normal model output, not through a separate exploit.

Retrieval Augmented Generation corpus exposure

A retrieval layer pulls chunks from a vector store keyed by the user prompt and pastes them into the model context. If the retrieval does not authorise per caller identity, a generic question returns chunks indexed from another tenant, another department, or a document the user never had read access to. The model summarises the chunk faithfully and the caller reads content they were never granted.

Conversation memory bleed across sessions or users

A shared cache, a misrouted session token, a poorly scoped memory feature, or a prompt template that concatenates prior turns from a global buffer lets a new caller read fragments of a prior session. Multi-tenant deployments are the highest-risk shape because one buffer bleed crosses a customer boundary.

Prompt-embedded secrets and personalisation fields

A developer pastes a customer identifier, a partner API key, an internal endpoint URL, a feature-flag list, or a jurisdictional flag into the system prompt for convenience. Any caller who can extract or paraphrase the prompt context reads the secret. The system prompt leakage page covers this surface as its own class and pairs to LLM02 when the leaked secret is itself sensitive data.

Embedding inversion and membership inference

A vector store exposes embeddings through a search endpoint, a debugging surface, or a public sandbox. An attacker reconstructs the source text from the embedding through known inversion techniques, or queries with candidate strings to learn which were in the training set. Both are inference-side disclosures the application did not consider when designing the embedding API.

Tool-call results and structured outputs

An agent calls a downstream tool, a database query helper, or a search service and embeds the raw result into its response. The tool returned more rows or richer fields than the model was supposed to summarise. The agent emits the structured payload verbatim and the caller reads the full record set.

Observability and logging destinations

Application logs, LLM provider traces, third-party observability vendors, and feature-flag evaluation services capture the prompt, the retrieved chunks, and the model response. The sensitive data is now in storage destinations the security team did not enumerate, with retention policies the team did not write, often outside the data residency the contract promised.

Public starter templates, debug routes, and AI gallery surfaces

A team publishes a starter prompt, a demo agent, a fine-tuned model snapshot, or a sample RAG corpus to a public gallery for a launch or a community contribution. The artefact carries internal context the engineering team forgot was sensitive: customer names, vendor relationships, internal product names, or unannounced features. The disclosure happens before any attacker exists.

Treating the model as a black box rather than a data handler

The team treats the LLM as if the data it sees evaporates at the end of the call. The deployment shape (memorisation in weights, indexed embeddings on disk, traces captured by observability) means the data persists in many places the application owner has to inventory and protect.

Fine-tuning on uncurated internal corpora

A team takes the internal documentation, the support archive, or the analytics warehouse and feeds it into a fine-tune without a minimisation pass. The model now reproduces fragments of those records to any caller. The fine-tune is the slowest path to redact because the data is now in the weights.

Retrieval indexes that ignore caller identity

A RAG pipeline is built with a vector store that returns the closest chunks to the query. There is no identity filter that asserts the caller had read access to the source document. The retrieval is correct but the authorisation is missing.

Personalisation by literal embedding in the prompt

A team embeds the user real name, account identifier, plan tier, jurisdictional flag, customer code, or internal field directly in the system prompt for convenience. The fields become part of the leakable surface every extraction prompt probes for.

Output post-processing as an afterthought

The output filter is a regex list someone wrote during the launch sprint. The list does not cover the customer identifier patterns, the internal endpoint patterns, the partner names, or the regulated data classes the threat model would have surfaced if it had been written. The filter passes the sensitive content through unchanged.

Trusting observability destinations the security team did not contract directly

Engineering teams adopt LLM observability vendors, prompt evaluation services, or feature-flag evaluation services without a redaction agreement. The sensitive prompt content, the retrieved chunks, and the model output now sit with vendors whose data retention, residency, and access controls the security team has not reviewed.

Automated detection

• SecPortal code scanning runs against connected GitHub, GitLab, and Bitbucket repositories and flags prompt construction sites that embed customer identifiers, partner credentials, or internal endpoints; fine-tuning data pipelines that ingest PII without a documented minimisation step; retrieval helpers that pass indexed documents through to the model without per-identity authorisation; and observability sinks that capture the prompt or response without a redaction step
• Authenticated scanning drives the LLM-backed endpoint with a curated extraction corpus under a real session: direct PII probes, cross-tenant identifier probes, retrieval-source probes (asking for the full source document behind a summary), conversation-memory probes (asking what the prior session contained), and intellectual-property probes targeting internal documentation fragments that should remain scoped
• External scanning discovers exposed agent endpoints, public chat surfaces, debug routes, public starter templates, leaked fine-tuned model snapshots, and public documentation that may have published the production prompt, the redaction policy, or the indexed corpus by accident
• Continuous monitoring re-runs the extraction probe on a defined cadence so a prompt edit, a model upgrade, a refreshed fine-tune, a new retrieval source, or a new agent tool that re-opens a previously closed disclosure surfaces against the baseline rather than waiting for the next pentest cycle
• Bulk finding import accepts the validated output of dedicated LLM red-team tools, AI safety scanners, or prompt-evaluation services so external probe results land on the same engagement record as the SecPortal-driven probes, with one CVSS 3.1 calibration applied across the LLM02 finding chain

Manual testing

• Produce the sensitive data inventory for the LLM feature: what personal data, what secrets, what intellectual property, what regulated data, and what cross-tenant data the feature touches on the request path, on the training data pipeline, and on the retrieval index
• Walk the training and fine-tuning corpus build pipeline and confirm a minimisation step exists, with a documented redaction rule per data class and an audit trail per training run that records what was removed
• Probe the deployed feature with direct PII extraction prompts that ask the model to recite known patterns, list known users, or recall examples from the training set, and record any verbatim or paraphrased response that confirms memorisation
• Probe the retrieval layer by asking the agent for the full source document behind a summary, for the metadata of the chunks it cited, for the document title or owner, and for cross-tenant identifiers that the calling identity should not have access to
• Probe conversation memory by opening a new session under a second identity and asking the model to recall details from the prior session, including arbitrary tokens the prior session contained and unique identifiers that would only appear in a leak
• Review application logs, LLM provider traces, observability vendor dashboards, prompt evaluation services, and debug dumps for any destination that captures the prompt, the retrieval payload, or the model response without a redaction step
• Walk every public surface the team has shipped (starter templates, demo agents, public model snapshots, sample corpora) and confirm none of them carry internal context, customer names, partner relationships, or pre-announcement product references

Finding with the extraction prompt, response, and content class

The finding captures the extraction prompt the attacker used, the model response, the substring that leaked, the upstream data source (training, fine-tune, retrieval, memory, prompt), and the content class (PII, secret, IP, regulated data, cross-tenant). AppSec, product security, and data security read the same record the engineering team uses to reproduce the disclosure and ship the fix.

Code scanning across LLM data handling sites

Code scanning runs Semgrep against connected GitHub, GitLab, and Bitbucket repositories. Findings surface at prompt construction sites embedding personalisation fields or partner credentials, fine-tuning data pipelines without a minimisation step, retrieval helpers that bypass identity scope, and observability sinks that capture the prompt or response without redaction.

Authenticated scanning with the extraction probe corpus

Authenticated scanning runs against the LLM-backed endpoint with a curated extraction probe corpus under a real session. Direct PII probes, cross-tenant identifier probes, retrieval-source probes, conversation-memory probes, and intellectual-property probes all execute, and each finding ties the response to the substring of the upstream data source that leaked.

External scanning across exposed AI surfaces

External scanning enumerates public agent endpoints, debug routes, public starter templates, leaked fine-tuned model snapshots, and public documentation that may have published the production prompt, the redaction policy, or the indexed corpus by accident. The finding ties the leaked content back to the public surface the team has to update or take down.

Continuous monitoring against AI feature drift

Continuous monitoring re-runs the extraction probe on the configured cadence. A prompt edit, a model upgrade, a refreshed fine-tune corpus, a new retrieval source, a new agent tool, or a refactored output filter that re-opens a previously closed disclosure surfaces against the baseline rather than waiting for the next pentest cycle.

Bulk import for external LLM red-team output

Bulk finding import accepts CSV intake from dedicated LLM red-team tools, AI safety scanners, and prompt-evaluation services. External probe results land on the same engagement record as the SecPortal probes, with one CVSS 3.1 calibration applied across the LLM02 finding chain and one owner assignment per finding.

Retest after the remediation ships

Once the fix deploys (the training data is re-curated and redacted, the retrieval layer is rewritten to filter by identity, the memory buffer is scoped per tenant, the output filter is extended, the redaction agreement with the observability vendor is in place), a targeted retest replays the original extraction probe corpus and records the post-fix response on the finding. The finding closes against the evidence rather than against a developer assertion.

AI-assisted writeups with verified scope

AI reports generate the writeup, the executive summary, and the developer-facing reproduction steps from the finding record. The narrative stays within the verified evidence on the finding (the extraction prompt, the model response, the leaked substring, the upstream data source, the content class) and does not invent guardrails, sandbox behaviour, or runtime tooling the product does not have.

Document management for the canonical inventory

Document management stores the sensitive data inventory, the redaction policy, the retrieval authorisation rules, the output filter list, the extraction probe corpus, the observability redaction agreement, and the regulatory mapping per finding. Each artefact attaches to the finding so the auditor reads the operating record the engineering programme runs against.

Compliance tracking pairs the fix to control evidence

Compliance tracking maps LLM02 findings to the controls that read against them: ISO 27001 A.5.34 PII, A.8.10 information deletion, A.8.11 data masking, A.8.12 data leakage prevention; SOC 2 CC6.1 logical access, CC6.7 transmission and disposal; PCI DSS Requirement 3 data protection; NIST SSDF PW.5 secure coding; NIST AI RMF Map, Measure, Manage; ISO/IEC 42001 AI management system; OWASP LLM Top 10 LLM02.