Misinformation in LLM Applications
detect, understand, remediate

Hallucinated identifiers and references

The model emits a CVE identifier, a case number, a legal citation, a function signature, a configuration flag, a vendor product name, a regulator decision number, or a research paper title that does not exist. The downstream reader treats the identifier as authoritative because the surrounding sentence reads correctly. The fabricated identifier then propagates into tickets, code, remediation reports, audit responses, and customer communications.

Fabricated source citations

The model attributes a claim to a real publication, agency, person, or document that does not contain the claim. The citation looks legitimate at a glance because the source exists. A reader who clicks through finds nothing relevant or finds the source contradicts the claim. The class includes invented URLs that look canonical but were never published.

Plausible-but-wrong instructions and policies

The model invents a configuration step, a CLI flag, a policy clause, a contract term, a settlement number, an opt-out URL, a regulatory deadline, or a process the organisation never actually adopted. The reader follows the instructions, makes the configuration change, signs the contract, accepts the policy, or reports the deadline to the customer, and the wrong content propagates downstream.

Hallucinated package and API names

A code-assistant feature recommends a dependency that does not exist on the package registry, an API endpoint the vendor never shipped, a SDK method that the library does not implement, or a config key the framework does not recognise. The class pairs to package-confusion supply-chain attacks where attackers later register the hallucinated name with malicious payload.

Confidently misread retrieval

The retrieval layer surfaces a real document, the model paraphrases it, and the paraphrase introduces a factual error the source never asserted. The error survives because the citation back to the source convinces the reader that the answer is grounded. The retrieval was correct, the grounding step failed silently.

Numerical and arithmetic hallucination

The model invents a percentage, a count, a calculation, a benchmark result, a price, a ratio, or a statistic that has no source. The error is hard to detect because numbers read as facts. Decisions taken on the basis of the invented figure (budget approvals, risk weighting, executive reporting, regulatory filings) inherit the defect.

Overconfident refusal misinformation

The model asserts that an action is impossible, a feature is unavailable, a policy forbids the request, or a regulation requires a specific posture, when none of the four is true. The user accepts the assertion and abandons a legitimate action. The class disproportionately affects customer-facing support agents and internal compliance assistants.

Domain-shifted hallucination across versions

The model was trained on documentation, APIs, regulations, or framework versions that have since changed. The answer reflects the older state with confidence, the user reads the answer as current, and a real change (a deprecated endpoint, a revoked control, an updated framework clause, a recalled product, a withdrawn standard) goes unaccounted for in the downstream decision.

Treating LLM output as deterministic text

The team mentally models the LLM call the way it models a function call: same input gives same output, output is correct unless input is malformed. The architecture inherits the assumption. The truth is the model is a probabilistic generator whose output distribution shifts with temperature, model version, prompt edits, retrieval payloads, and context. Every reliability control has to live outside the model call.

Retrieval mistaken for guarantee of grounding

A RAG pipeline is added and the team assumes the model now answers from sources. In practice, the model freely mixes retrieved content with parametric knowledge, fills gaps with confident invention, and ignores low-quality retrieval. A retrieval step without enforced grounding and citation is not a guarantee of factuality.

Confident-tone prompt instructions

The product team asks for a confident voice because hedging reads as weak. The prompt rewards confidence in language, the model produces confident language for both correct and incorrect answers, and the user loses the ability to calibrate trust on confidence alone. The brand voice goal collides with the reliability goal and the reliability goal loses by default.

No representative evaluation set

The team has no curated set of (prompt, expected behaviour) pairs that represents the answer types the feature is meant to handle, with ground-truth labels and edge-case coverage. Without the set, hallucination rate cannot be measured, regressions cannot be detected, and the only signal is a customer complaint after a wrong answer reaches production.

Logging captures the request but not the verification

Application logs record what the model returned and which user received it. The logs do not record whether a citation was checked, whether the cited source contained the claim, whether the user accepted or corrected the answer, or whether a downstream action executed against a hallucinated argument. Post-incident reconstruction defaults to grep against unstructured text.

Schema-free pipes between model and downstream systems

The model returns natural-language text into a downstream parser that accepts whatever arrives. Identifier fields, numerical fields, status codes, and tool arguments are not validated against a registry, a range, or a lookup. The hallucination passes the parser because the parser was permissive by design.

Automated detection

• SecPortal code scanning runs against connected GitHub, GitLab, and Bitbucket repositories and flags LLM call sites that emit text into a security-sensitive downstream sink (code generation, contract drafting, ticket creation, financial calculation, regulatory filing) without a paired verification step, a citation requirement, or a schema constraint on the structured fields
• Code scanning also flags prompt-construction sites that instruct the model to answer confidently, suppress hedging, or refuse to admit uncertainty for answer types the application cannot independently verify, since the confident-tone instruction is correlated with downstream hallucination harm
• Authenticated scanning drives the LLM-backed endpoint with a curated misinformation corpus (questions designed to elicit fabricated identifiers, fake citations, hallucinated package names, invented configuration steps, false numerical answers, overconfident refusal, version-shifted documentation) under a real session, and records every response whose factual claims fail a verification check against the ground-truth source
• External scanning discovers public agent surfaces, public chat dashboards, debug routes, and public starter templates that may expose the feature to unauthenticated probing for misinformation, so a hallucination on a public surface lands as a finding before an external researcher writes the post
• Continuous monitoring re-runs the misinformation probe on the configured cadence so a model upgrade, a prompt edit, a retrieval-pipeline change, a fine-tune update, or a context-window resize that regresses hallucination rate shows up against the baseline rather than waiting for the next pentest cycle or the next customer complaint
• Bulk finding import accepts CSV output from third-party LLM evaluation frameworks (Ragas, DeepEval, OpenAI evals, internal harnesses) so the engineering programme can land per-test-case hallucination findings on the same workspace where the rest of the security backlog lives, with the test-case identifier, the asserted answer, the ground truth, and the divergence captured on the finding record

Manual testing

• Build a representative test set of prompts that reflect the answer types the feature is meant to handle, with ground-truth labels, citation requirements, and edge-case coverage including questions whose correct answer is "I do not know"
• Run the test set against the production prompt, model, retrieval, and tool registrations and measure the hallucination rate, the unsupported-citation rate, the unjustified-refusal rate, and the calibration of model-stated confidence against verified correctness
• Probe for fabricated identifiers (CVE IDs, case numbers, function names, package names, regulatory references) with questions that have a known empty result, and record any response that invents an identifier rather than admitting absence
• Probe for unsupported citations by asking questions whose answer requires a source the retrieval cannot supply, and verify whether the model fabricates a citation or correctly declines to cite
• Probe for numerical hallucination by asking for percentages, counts, prices, ratios, or statistics that have no canonical source in the retrieval, and verify whether the model invents a figure or correctly states the absence
• Repeat the full test set on every model upgrade, prompt edit, retrieval change, fine-tune update, and tool registration change as a release gate, and treat regression on hallucination rate as a release-blocking defect

Finding with the prompt, the wrong answer, and the ground truth

The finding captures the prompt the user submitted, the confidently wrong output the model returned, the ground-truth source the answer should have grounded against, and the downstream action (or attempted action) the wrong answer triggered. AppSec, product security, AI engineering, and ML platform read the same record the engineering team uses to reproduce the defect.

Code scanning across prompt and verification call sites

Code scanning runs against connected GitHub, GitLab, and Bitbucket repositories. Findings surface at LLM call sites that emit to a consequential downstream sink without a paired verification step, at prompt-construction sites that suppress hedging, and at structured-output sites that pipe model text into a parser without schema validation. The remediation lands at the construction site, not at a perimeter filter.

Authenticated scanning with the misinformation probe

Authenticated scanning drives the LLM-backed endpoint with a curated misinformation corpus under a real session. Fabricated-identifier probes, fake-citation probes, hallucinated-package probes, invented-configuration probes, numerical-hallucination probes, and overconfident-refusal probes all execute, and the finding ties each response to the divergence between the asserted answer and the verified ground truth.

External scanning across exposed AI surfaces

External scanning enumerates public agent endpoints, public chat dashboards, debug routes, and public starter templates that may expose the feature to unauthenticated probing for misinformation. The finding ties the hallucination on the public surface back to the access path the team has to gate.

Continuous monitoring against eval regression

Continuous monitoring re-runs the misinformation probe on the configured cadence. A model upgrade, a prompt edit, a retrieval-pipeline change, a fine-tune update, or a context-window resize that regresses hallucination rate shows up against the baseline rather than waiting for a customer complaint, with the changed hallucination rate, the affected answer types, and the trigger captured on the finding record.

Bulk import from external eval frameworks

Bulk finding import accepts CSV output from Ragas, DeepEval, OpenAI evals, and internal harnesses so the engineering programme can land per-test-case hallucination findings on the same workspace where the rest of the security backlog lives. Each finding carries the test-case identifier, the asserted answer, the ground truth, the answer-type tag, and the regression baseline.

Retest after the remediation ships

Once the fix deploys (the retrieval-grounding requirement, the citation-enforcement step, the post-generation factuality check, the schema constraint on structured fields, the uncertainty-surfacing UI change, the prompt edit that permits "I am not sure", the eval added as a release gate), a targeted retest replays the original misinformation probe against the new construction and records the post-fix response on the finding. The finding closes against the evidence rather than against a developer assertion.

AI-assisted writeups with explicit honest scope

AI reports generate the writeup, the executive summary, and the developer-facing reproduction steps from the finding record. The narrative stays within the verified evidence (the prompt, the asserted answer, the ground truth, the answer-type tag, the eval harness identifier) and does not invent guardrails, factuality services, or runtime tooling the product does not have.

Document management for the eval harness record

Document management stores the representative test set, the ground-truth labels, the answer-type taxonomy, the citation policy, the schema definitions, the uncertainty-surfacing rules, and the human-in-the-loop policy. Each artefact attaches to the finding so the auditor reads the operating record the engineering programme actually runs against.

Compliance tracking pairs the fix to control evidence

Compliance tracking maps misinformation findings to the controls that read against them (ISO/IEC 42001 AI management system, ISO 27001 A.5.34 privacy and protection, A.8.16 monitoring activities; SOC 2 CC4.1 monitoring and CC7.2 system monitoring; NIST AI RMF Measure 2 and Manage 4 functions; NIST SSDF PW.5, PW.7, and PW.8; OWASP LLM Top 10 LLM09).