Misinformation
LLMs confidently state false things. When the output drives a decision, the false thing becomes a bug, a liability, or a breach. Misinformation in the LLM Top 10 is about output you trusted that was wrong.
01What it is
LLM outputs that present incorrect, fabricated, or misleading information as authoritative. Causes include hallucination, training-data error, RAG retrieval of stale or wrong documents, deliberate prompt injection, and confident over-extrapolation in low-resource domains.
02Why it matters
Generic AI chat is forgiving — users discount answers they verify. Production AI integrated into customer support, financial advice, medical triage, or legal research is not. A wrong answer becomes a service failure, a regulator complaint, or a tort claim. In regulated industries, the cost is uncapped.
03Attack vectors
- Hallucination — model fabricates citations, numbers, names, regulations.
- Over-extrapolation — model answers confidently in domains it should refuse.
- Stale RAG — retrieval returns outdated policy text the model presents as current.
- Source confusion — model conflates two retrieved chunks, attributes one's claim to the other's author.
- Adversarial-framing — attacker phrases the prompt to elicit confident wrong answers.
- Cross-language drift — answers degrade in lower-resource languages, including Indian languages.
04Defence patterns
- Grounding-only architectures — model is only allowed to answer from cited retrieved text. Refuse otherwise.
- Cite sources — every claim links to the retrieval chunk. Users (and auditors) verify.
- Eval harnesses on domain-specific golden sets — fail the deploy if accuracy drops.
- Confidence + abstention — train or prompt the model to say "I do not know."
- Human review for high-stakes outputs — medical, legal, financial advice never auto-published.
- Output classifiers for factuality (LLM-as-judge, retrieval-grounded factuality checks).
05Detection
Signals to watch
Sampling-based audits. User-facing "report wrong answer" flow that triages fast. Production drift detection — sudden change in user complaint rate.
06India context
DPDP · RBI · CERT-In
DPDP Act 2023 requires accuracy of personal data; an LLM that "knows" wrong facts about a person and presents them as truth is non-compliant. For BFSI customer-facing AI, mis-advised customers create both regulatory and reputational risk. SEBI investment-advice rules apply if outputs cross into financial advice.